Nah Rich you're right, if someone slates you for that then they're a bit weird. When I was in primary school a boy called me a "nigger", his mum was called into school and she said "Is he actually black?" the reply was obviously no, so she said "It doesn't matter then does it." What the f**k is that? The words an insult no matter who it's said to. It can't be just against black people.
Then a extremely disruptive child at the school my mum works at (same primary school as mentioned before) is, "Half cast" (sorry if that affends anyone) and his mum says he's only naughty because the other children bully him because he is a "half cast". She says it everytime he gets in trouble, and because the head teacher is currently as soft as fresh curry like poo, he gets let off everytime. Racism card for the win, I might use it.