Yes. Doing so makes you a hypocrite. Don't worry through, there's no shortage of hypocrisy in America. It's practically a requirement to be at least unwittingly hypocritical. Just by drinking Coke or tipping a waiter you're contributing to a broken system designed to exploit people for maximum profit.
But here's the rub. You can't, in any practical sense, escape that crap, however, you can choose to not deliberately contribute to stuff outside your immediate wheelhouse. It's one thing to buy a chocolate bar out of a vending machine, but investing in Nestle? That's a choice, and one you could have easily skipped. You could skip the candy too, but it's very, very hard (and impractical) to refuse every corporate product ever. Everything, from the materials in your electronics to your mortgage company, to most food from lettuce to frozen chicken, exploits people. But you don't have to voluntarily make the problem worse.
And on the sliding scale of morality, investing in slavery - in this case the prison industrial complex is just greed and indifference to the cost in human suffering. Seriously research it, slavery in all but name has been part of the plan since the Reconstruction era after the Civil War. We never had a justice system; we have a punishment system that hungers for the labor of the downtrodden, especially of minorities.
So if you want to at least try and be a better person, and investing is something you want to do, look into the companies you're investing in. See what their executives are paid compared to their workers here and abroad. There are companies that you can ethically justify investing in - small companies, co-ops, credit unions, pro-union companies, companies actually trying to solve problems or make the world better, like solar manufacturing, etc.
If you want to invest in human suffering, then you're going to have to make peace with being a bad person and being judged for it. I'd advise at least trying not to. It's a hopeless battle, but fighting honorably is its own justification.