To be fair, wouldn't an admin always be able to see your downvotes, even without the current completely open view of it? Admins in general usually get information you're not privy to on sites.
That said, your best bet would be to just find magazines that offer the same thing, just not run by him. Or make a new one. If he keeps it up you'd likely get traffic.