Friends wanting to hike with me

Several friends have expressed an interest in hiking a day with me as I travel close to them along my thru... I said "yeah we do that" but definitely hesitated. I hate to be rude, especially with people who are supporting me. But I'm picturing them slowing me down or otherwise interfering with my experience. Is the hesitation just my anxiety or do I have genuine concerns here? What do you think would be a good way to handle this scenario without creating resentment? TIA hikers!