Definitely agree that the free market can come up with some undesirable solutions which is where I think regulation comes in to "guide" the more desirable outcomes that can be found organically. Personally, I think maximum occupancy and increased supply should be the goal where there is limited supply like NYC. Things like a vacancy tax and better zoning could help a lot.
Also, I'm not sure that I trust the government to find the best use of space either, especially in the face of corporate lobbying. The current road/highway system being built to the detriment of public transportation is a good example where a government prescribed solution can have a negative impact decades later.
Definitely not advocating for full blown free market capitalism. My comment was more along the lines of letting the market organically find the best solutions. The government should set broad goals, like "maximize the amount of occupied housing units and minimize homelessness" and then provide the appropriate incentives to guide the market in that direction.
I agree that for inelastic goods like healthcare, food, water, shelter the situation is even more tricky. NYC just seems to be limited in that sense with already high density and low supply. Having any form of vacant units should be taxed heavily. Maybe even extend this to progressively tax larger units that reduce density. Billionaires row where the ultra wealthy have an entire floor for an apartment that they never use makes no sense to me.