But, if you're describing people you'd use any distinguishing features, such as hair colour, build etc. Why should it be wrong to use someone's skin colour to describe them?
It's when people treat others differently because of their skin colour, then you have a problem. If it's just for description then I see no problem.