I disagree, you'll have your backups, so even if everything breaks you will have a failsafe. If you get compromised it's still not an issue: Everything server side is encrypted, the safety is in the clients and your master password length.
So, I see no particular differences with other services. Considering I hear of some issues with bitwarden servers that are constantly under attack, selfhosting could even increase the availability.
ssh -p 12345 would leave your boxes accessible from anywhere too. Other blocks of IPs receive 10 times or more requests, as scanners can focus on blocks of ips from major providers.