Actually the history of that word is much longer and more convoluted than that. You can just google it. I should probably find actual books when I get the time. But anyway it isn't quite as you are describing it, the true prerogative nature of the word is lot more recent creation than slavery times. And even if you are right, again I ask why should we continue to give a power to a stupid word? Words are powerless and only have power that we give it to them. As for other cultures well good for them I guess.
Also point is that he wasn't using it with racist intentions. Just as a random swear word. He might swear every day, who knows, but at the end of the day it was not said with racist intentions. Also he apologized and the whole stupid thing has been over for ages so why are we even digging out old shames?