I've been using Emacs w/ TRAMP for remote dev for 3 years. I use lsp-mode w/ clangd, eshell, magit. Because of my setup I can't build locally, both as the project won't build on macOS and because it'd probably take 45m-1hr to clean build versus a few minutes on the server.
I agree that the experience isn't flawless, despite using ControlMaster etc I still have latency on save/revert, and staging+committing in Magit too. The worst is getting the ssh sessions "crossed up", where lsp-mode is transferring some data or a grep job is running and I hit C-x C-f, and Emacs locks up, needing a healthy dose of C-g spam to get it back.
Despite all the problems, I still feel that Emacs is a better remote dev environment for me. I work on a large archaic project with an elaborate build system, multiple "in-house" languages, and Emacs works wayy better for weirdo projects like this than VS Code. The rest of my team, and the company as a whole, uses VS Code and I've seen coworkers have it drop out, lock up, and be clunkier than TRAMP in some situations. The amount of configuration to get it working well (especially clangd+lsp) isn't any easier than Emacs.
IMO all that Emacs needs to let it stand up against VS Code for remote dev is to improve the performance and polish (latency and lockups). The architecture is fundamentally the same as VSCode, especially if you're using lsp-mode or eglot. Emacs will never appeal to the "Out-of-the-box" users as VS Code does, but for the power users it should still be fast and work seamlessly.
I may have a few tweaks I can check tomorrow at my work machine, but mine was never that bad. I find the diff is slow and can freeze up for a minute or two if you run a huge multi thousand file diff, but I don't imagine its that. Have you tried disabling magit-revert mode, or eglot mode? LSP or auto-revert checking the file state are usually culprits of lag for me.