You don't want the statue taken down? Fine. That doesn't make you a white supremacist. I'm sure people have tons of reasons why they might not want a statue taken down. Some are probably non-racist.
You march with people chanting Nazi slogans? You are the company you keep in this situation. And you are not a very fine person.
Don't misconstrue me to be saying people don't have a right to free speech. Personally I prefer when people admit to being neo-Nazis and white supremacists. I wish they didn't exist. But I'd rather know someone is a piece of shit when they are one.