I agree to some extent, as there are plenty of distros that don't do anything significantly different from each other and don't need to exist. I also see what you mean about desktop environments. While I think there's space for all the small exotic window managers that exist, I would say we probably don't need as many big fully integrated desktop environments as there are now. (Maybe we should have only one aimed at modern hardware and one designed to be lightweight.)

That being said, there is plenty of duplication of effort within commerical software too. I would argue that if commercial desktop GUIs currently offer a better user experience than Linux desktop environments it's more in spite of their development model than because of it, and their advantage has mostly to do with companies being able to pay developers to work full time (instead of relying on donations and volunteers).

There are a couple reasons I think this:

In a "healthy" market economy there needs to be many firms that offer the same product / service. If there is only a small number (or, worse, only one) that performs the same function the firm(s) can begin to develop monopolistic powers. For closed source software development this necessitates a great deal of duplicated effort.
The above point is not a hypothetical situation. Before the rise of libre software there were a ton of commercial unices and mainframe operating systems that were all mostly independently developed from each other. Now, at least when it comes to running servers and supercomputers, almost everyone is running the same kernel (or very nearly the same) and some combination of the same handful of userspace services and utilities.
Even as there is duplication of effort between commercial firms, there is duplication of effort and wasted effort within them. For an extreme example look at how many chat applications Google has produced, but the same sort of duplication of effort happens any time a UI or whole application is remade for no other reason than if the people employed somewhere don't look like they're working on something new then they'll be fired.
Speaking of changing applications, how many times has a commercial closed source application gone to shit, been abandoned by the company that maintains it, or had its owning company shut down, necessitating a new version of the software be built from scratch by a different firm? This wastes not only the time of the developers but also the users who have to migrate.

Generally I think open source software has a really nice combination of cooperation and competition. The competition encourages experimentation and innovation while the cooperation eliminates duplicated effort (by letting competitors copy each other if they so choose).

[–] thawed_caveman 2 points 2 months ago

I'm going to add on to this: aside from every competitor having to build their own product from scratch, a vertical structure of leadership allows for an individual or small group to have massive resources invested into their ideas. Which are so very often deeply flawed ideas.

Have you looked at Paint3D? Microsoft had a vision for a future where we interact with 3D objects in daily life and we would need a basic software to edit them, just like we use Paint to edit the images that we interact with. It's kind of insane, i can't even guess how many man-hours were spent on it by Microsoft developers.

And absolutely nobody has ever used it.

You can find more extreme examples, especially if you look outside of software (oh god the military); but Paint3D is one of the more accessible ones, any Windows user can check it out right now.