Selfhosted

Modded minecraft servers are heavily dependent on single threaded performance. For more vanilla servers paper helps a lot. For forge I highly recommend trying mohist. It isn't compatible with all forge mods but it works well enough that you can just replace the server jar in many modpacks and see a large performance boost.

The biggest thing that slows down mc servers in my experience is world gen. Pre generating the world and adding a world border can help a lot.

I've not done a larger scale fabric server so I can't offer much advice in optimizing it but the client speed ups available through fabric look very impressive.

If you are running a server without world borders or with a lot of simultaneous players I'd look in depth on what ssd you're saving the world to. You want dram cache, random write speeds are way more important than sequential. If you can find an Intel optane for cheap they are pretty amazing. The ssd is less important than your cpu and having enough ram to run the server.

Generally an older gaming pc is better than an older server. Again you are targeting single threaded performance. If you are purchasing hardware it might make more sense to go with lower end new hardware than higher end old hardware. It's all about trade offs for your use case and budget. For a long time I just used my main pc to play games and host servers (ram is cheaper than another pc) but I tinker too much to keep good 'server' uptime.

Transcoding can get pretty taxing on a system but any semi modern quad core can handle a few 1080p streams or a 4k stream. Plus you can use a gpu for transcoding. The nice thing is it scales with core count pretty well so older server or workstation hardware works well.