I read an interesting blog on this: http://beyondthekerb.org.uk/2016/03/23/an-obvious-problem/
"Yet jurors, regardless of how well they drive themselves or even whether they drive at all, are assumed to be capable of adopting the viewpoint of a competent and careful driver in order to decide what is or is not obvious to them.
And this gives us our de facto definition of “a competent and careful driver”: it is a person chosen entirely at random."
Another interesting read, not directly related but more generally about jury decisions: http://beyondthekerb.org.uk/2015/06/19/karrs-choice/
Putting the two together, the reason dangerous driving charges fail is because the jury is made up of average people and the average person believes driving to the ideal scenario is competent and careful*. The collision wouldn't have happened if the cyclist hadn't made an error, or the wind hadn't blown, or the road surface wasn't poor so it wasn't the driver's fault, regardless of how close or how fast he was, or how little he could see.
The poor conviction rate isn't the problem, its simply one symptom of the underlying problem of what the average person considers safe driving around cyclists. You could try and solve the problem by fighting to reform the justice system and increasing conviction rates. You could also do it by building infrastructure and making the average person a cyclist. Build it well and it has the added benefit of removing the conflict altogether.
* This is why assertive cycling, taking the lane, etc makes such a big difference. It prevents the average person gambling with your life by making it more obviously dangerous.