Does the microphone on the xm4 is really bad during phonecalls or is it just mine ?

I've had the xm4 for like 2 years but everytime i'm in a phonecall with someone, they hear every sound around me pretty loud. So I have to turn it off or speak directly to my phone so that they can hear me correctly

Am I the only one ?