My case chiefly rests on his years of white supremacist rhetoric — which you didn’t address. I mentioned those other two bits at the end of my comment. On their own, I agree, they are slightly concerning but not damning; but in the context of his show’s content and rhetoric, they are.
Racism is the -ism of race — the idea that race is one of the (or the) most important traits of a person.
That through race, you can assume things about them like their worth, how you should treat them, how they act, feel, etc.
The way I see, all of these -isms are all like that (sexism, nationalism, etc).
So no, it’s not racist to call yourself white. It would be racist, though, to say you’re worth more or less than others because of your being white.