How should a partner react to your anxiety?

I am in "recovery" from pretty bad agoraphobia due to fear of panic attacks. It was hard to drive 5 miles away from my house. I have been talking to a guy for a while now. He usually drives to my place, but has said that I'm not showing much effort since I am never coming to him. I told him I have anxiety and that's why but he never asked anything further. I've been extremely depressed and lonely the past several months. He came over a couple days ago and we cuddled and it was nice. I felt really good that we had such a good time. He wanted me to come his way next so today, I mustered up the courage to drive to him. He lives 30 miles away. 30!!! I drove 30 miles to see him and did NOT have a panic attack. However, when I got to his place, I was feeling very anxious and I told him I was anxious. He said he didn't want me to be uncomfortable however as the day went on he seemed to get frustrated... It was hard to think and do things normally as I could since my mind was elsewhere. For example, when we went to get food, I almost got out of the car without taking my keys out of the ignition. I said, "I don't know where my mind is today" and he told me to slow down. Then, later, he was giving me directions to get back to his house and I missed a turn and I said, "I'm really out of it today!" and he rudely replied for me to wake up. I got really upset and this started to put me on the verge of a panic attack. I was trying hard to maintain a peace of mind away from being anxious while we were together but when I started to frustrate him, my anxiety would grow more. He apologized right after once I started to cry but when we got back to his house, I felt really uncomfortable and told him I wanted to leave. We haven't talked much since. I ended up texting him telling him I was sorry I left, but really I am more upset with how he started to handle me when my anxiety was affecting the way I was acting. I don't know how he'd react to a panic attack. He is out with friends now. I feel stupid. I tried mustering something that I really didn't want to do in hopes it would turn out for the better, but now things are weird and I feel exposed and embarrassed. How am I going to continue dating like this? I feel like a fucking burden. Thanks for reading if you got this far.