Grauniad article on commuter drafting
Not knowing they're there makes it a little worse in my opinion, because when you're riding on your own you might do things differently. For example on a certain bike rather than riding round potholes I'll bunnyhop them. If I knew someone was behind I'd be a bit more obvious moving round it, and maybe even do the proper hand signal. But bunnyhopping means the person behind may plough into it and come off.
But the big thing is the sudden need to brake. For whatever reason, form people pulling out, to red lights. I'd rather not end up with someone then riding into the back of me because they'd been drafting, and take us both down.
Also there's just basic politeness. Last week I had a guy drafted me for about 2 miles into a headwind, clearly just having a breather, because he then went past (on a straight of about a mile) and just powered away. A little bit of reciprocal help would have been nice.