My simple understanding is that fearing/disliking foreigners is xenophobic, fearing/disliking people because of the colour of their skin/national origin is racist.
Additionally there are other problems related to gender/sexual orientation/class/religion/other(ness).
Being upset because your town/neighbourhood is being swamped/overrun/taken over by 'new people' is not (necessarily) about racism or xenophobia. Is just another variation on Nimbyism.
If you live in a nice Green Belt village, the chances are that you'll object to a new housing estate (it might well be that it's almost identical to the one you've been living in for the last 20 years). If it's a council estate (unlikely these days) or intended for refugees, you might fight it harder - or stop objecting (as long as the school, health service and active travel provisions are built in).
It seems that "immigration" played a large part in many people's voting decision on Thursday. Sadly this is because for many years politicians - not least in the Labour Party - have (it would seem) chosen to not look too closely at whether 'concerns' were real, justified, misunderstandings, unpleasantness or tough.
This failure has allowed UKIP to define the agenda and attract many people who believe (rightly or wrongly) that they have been adversely affected and/or are not being listened too.
Of course UKIP doesn't have 'the' answer (not sure it has any answers to anything - except in the most glib and sloganistic sense) - and neither does the Brexit end of the Tory Party.
No-one has ever said 'Brexit will mean the end to all immigration' - or even "net migration".
It's far from clear what 'controlling our borders' and 'an Australian style points system' could/will mean.
The only 'promise' was the one about reducing (net) migration (or was that just the non-EU sort) to tens of thousands. So that's by the current Gov and all its MPs.
Now of course Nigel and Boris are saying we'll (only) let in the best sort of people - 'the ones with skills the UK needs' - 'Doctors, nurses, IT specialists' etc., not clear about care workers or fruit/vegetable pickers.
Of course (assuming there is still net migration - economic meltdown might sort that) it's not clear where they will be expected to live in this 'full up'/'overcrowded' island. Not clear if any of them will be allowed to be ill or have children.
No news on extra money for communities that have had significant numbers of extra people moving in lately. (If that ever happened, might it included areas with mass influx of students with their alien ways or yuppies with strange tastes in food and excesses of cars/bicycles?)
posts
