I got similarly wound up by a throwaway remark from a pedestrian a few weeks back, was doing the quick right our of Gilmour Place, navigating the potholed junction outside the Kings then directly left into Valleyfield.
There's a lot for your hands to be doing, braking, signalling, turning around potholes etc., checking over shoulders for cars or cyclists cutting up your inside or outside yadda yadda.
As I turned into Valleyfield, chump and his ladyfriend were stepping out onto the raised roadway bit on t'uther side of road and he loudly commented "where's your signal."
I debated either giving him a shout of a few short, sharp phrases or stopping and getting off to give him a lecture that A/ I did signal my turn before I made it B/ he couldn't see it because it was hidden behind my body C/ at that junction I'm more concerned about traffic behind me seeing the signal D/ we were nowhere near eachother E/ there's more pressing matters when making that turn than holding an arm out wide to the left as you go round the corner and F/ he's embarrasing his girlfriend and should try getting on a bike and shedding a few pounds.
But I went home silently boiling with rage about smart retorts I could have given and how he had spoiled an otherwise nice ride home.
I suppose the difference with such comments directed at cars is that we actually hear them. I think as these comments are either made to imrpess the friend or to deliberately rile you, the best response is either no response or to laugh it off and try not let it get to you.
Like you said, we can't win.
Amir and I were passed by an idiot in a Ford Focus on an otherwise completely empty A708 just before the Grey Mare's Tail. It's 2 lane, straight road with great sightlines. We heard the car coming, singled up and pulled in a bit only for the passenger to be hanging out the window yelling "why don't you pay road tax". I managed to respond with some anglo saxon while they were still in hearing distance but sadly there were too many miles in our legs to catch the car on the climb out of the valley and deliver the lecture about "road tax". Wish we had stayed side-by-side taking up a whole half of the lane now, just to have really incovenienced him. They obviously saw us in the distance and mate said to driver "ha ha, watch and learn, this is how clever I am".