"Being" one is perhaps taking it too far. How does one prove that someone is racist unless they actually act in that manner or at the very least incite racist violence? I'm not saying that because I endorse or tolerate racism in any way, but judging people about what they think or what we believe them to be is a slippery slope.
For example, let's suppose an "old timer" tells a black man that "we were better off when you people knew your place". It's obviously obnoxious. But is it enough to get someone jailed? I don't know. But of course when you have bona fide white supremacists infiltrating school boards and town councils, the point becomes moot. It took several years and an actual high-profile murder case to get the Neo-Nazis banned in my country, but at least it happened. It doesn't seem likely that a similar ban will happen in the US without some similar tragedy.