What would happen is not dentists installing your software themselves, without support.

What would happen, and already happened countless times is that you see companies appearing that will sell services to install and maintain your open source software. It will be done by software developers just like you, except they can spend their whole time working for their customer instead of working on improving the software itself.

You will soon realize that you make money from the same activity as them, except you also have to maintain the software (and spent time to build it in the first place).

Of course you can use shady techniques to make it harder for others to understand your code, or release the open source version with a delay, but if you play by the usual rules of open source, with open development, third party developers will be able to acquire as much knowledge as you about the product.

Not necessarily. The original developer might know the code base better, but the hostile fork needs no development cost.

The 3D printing industry is a really good example of something like that. Take for example Marlin. That's the firmware that runs on the vast majority of 3D printers. It is open source and thus freely available.

Many 3D printer manufacturers just download a copy of Marlin, change the logos and slap it onto their printer. They are never going to update it, because Marlin is great as is and most customers won't ever update it anyway. So the 3D printer manufacturer has a development cost of maybe 1-2h to dial in the config and replace the logos. Compared to the developers of Marlin, who spent whatever the time equivalent to 19691 commits is. Also, the 3D printer manufacturer earns money from the sale, the Marlin devs don't.

Or take a look at e3d, who recently stopped open sourcing their hardware. They created the E3D v6 hotend, which is by far the most common hotend to this day. The issue is, most people don't buy an original E3D v6, which costs ~€50-70 (the more common 1.75mm version), but instead they buy the cheap €15 copy from Amazon or the €5 version from Aliexpress.

Again, the copyists might not have the expertise that e3d put into their work, but they know the exact dimensions and the material that needs to be used, and they just make a perfect replica. No research costs, only manufacturing, and be done with it.

Open source is only sustainable if you get your money for developing, not for selling the product. So for open source to work, you need e.g. a Patreon or cooperate sponsorships to fund the development, so that after development is done (before any product is sold) you got your money.