I'm surprised no one seems to have mentioned the Paradox of Tolerance. Essentially if you tolerate intolerance, the intolerants will eventually seize power and make an intolerant society, the only way a society can become truly tolerant is by being intolerant towards intolerance.
It's paradoxical, but makes absolute sense. If you allow Nazis to spread their ideology eventually there will be enough Nazis to be able to take the power by force, and when they do they'll setback all of the tolerance that was advanced. The only way to prevent it is by cutting the evil at the root and prevent Nazis from spreading their ideology.
Personally I believe that punching a person who hasn't tried to attack me or anyone is wrong. But the moment someone openly preaches that someone else must be exterminated they're inciting violence which can encourage others to act on it, to me, morally speaking, attacking that person is as much self defense as if they were commiting the act themselves.
Would I personally punch a person because they're spewing hate? Probably not, I would probably try to talk to them and understand their point of view and try to convince them otherwise, since I believe that punching them would make the person close himself to any reasoning from outside of his group, which would make him more Nazi than before. But I also don't think it's morally wrong to do so, it's just not the optimal way of dealing with it.
Thanks for the correction, I've edited the comment, indeed that could be missinterpreted.