Politics is inevitable since it is literally making decisions in groups (along with all the other baggage attached to that including power structures, etc.). If you have a group of people you have politics. End of story. Anybody saying "keep politics out of my " is exercising politics: specifically the power structure elements of it. They are being oxymoronic (as well as the same minus 'oxy').
So it's not politics that's objectionable, I'm going to guess, but rather ...
... drum roll ...
PARTISAN politics.
And "partisan" doesn't mean "left/right". It means any kind of politics that separates and generates an in-group and an out-group. (The root "part" is in partisan as well as political party for a reason.) Partisans by their very nature automatically hate the political opinions of their non-partisans—their "out-group". (And people like me hate all partisans for being that way.) This is true if you're "left" (by whatever definition you use), "right" (ditto), feminist, religious, plutocratic, anti-religious, liberal (in the non-American sense of the term), progressive, racist, etc. etc. etc. If you're dividing society into parts, some of whom are deemed "not one of us" you're a partisan.
It is partisanship that is toxic for the most part. Partisanship is what gives you "the party of 'NO'" in most democracies (especially the USA, but not exclusively): you can't, as a partisan, admit that the "other team" did something good or has a good idea. This gives us ridiculous situations like the most memorable one to me: Pierre Trudeau lambasting Lester Pearson's plan to introduce wage and price controls, getting elected on a platform against that, then promptly introducing wage and price controls. It turns out Pearson's plan was a good one, but for reasons of partisanship Trudeau couldn't admit it. He was a "Liberal" (read: center-left by American standards) not a "Conservative" (read: center-slightly-left by American standards—keeping in mind that the American "left" party is a center-right party in almost any other country).
And indeed, I'll go a step farther: political parties are explicitly anti-democratic. They are specifically there to undermine democracy. In Canada, for example, you in theory vote in your riding to send a representative to speak on your interests to Parliament. Political parties subverted this, however, and now you vote for a party to provide a representative that represents the party's views to you. Your views don't matter.
So politics is inevitable. You can't avoid it. But partisanship doesn't have to be, and it's partisanship that is destructive, even when it's for a "good cause".