As much as white people hate to hear this, there is a difference between white people making a comment like that about a black person, and vice versa. I know that seems unfair, and maybe someday it will be, but right now we live in a world where we white people are vastly over represented in positions of power, wealth, entertainment, and politics. As such, we enjoy something known as "privilege". This is not an opinion that I'm pulling out of the air, it's a well documented sociological concept. Feel free to look it up.
So yea, a black guy can make a joke about getting shit for wearing a white guy's shirt. Black folks are also the only ones who are allowed to say "nigger" to each other. The relationship between blacks and whites is inherently unequal in this country right now. At some point we may truly live in a post racial society and that won't be true, but for now, it is.
And if that bothers you, just enjoy the fact that as a white person, you are more likely to be accepted for a loan with identical credit than a black person, more likely to be hired for a managerial position with equal qualifications than a black person, and much less likely to be stopped by the police for a random "stop and frisk" than a black person.
(Again, those aren't opinions, they're statistical facts that I'm pulling straight from my college sociology texts.)
My motto was always to keep swinging. Whether I was in a slump or feeling badly or having trouble off the field, the only thing to do was keep swinging. -Hank Aaron