It's kind of funny, having the calves slaughtered to get the milk that is naturally meant for them is considered vegetarian (as long as you personally don't eat the veal).
If they're kept on abusive factory farms, that's still vegetarian.
When the dairy cows gets their throats slit because milk production drops below profitablity after ~5 years, the milk is still seen as vegetarian (as long as someone else buys the meat).
No matter how much death and suffering takes place at the farm, the milk is seen as vegetarian. But at rennet, that's where they draw the line.
I think most people would actually consider factory farming unethical, they just put the blame on the producers for treating animals like shit. And the producers are locked into a race to the bottom for competitive prices, so they'd blame the customers/market conditions.