I'm a little late to the party, but a fairly easy way to combat that fear is to install Tailscale (free) on your server and have them do the same on the AppleTV. It's supported from version 17 I believe. It uses Wireguard to encrypt so your ISP won't see squat.
I'm hosting a server for friends and family too and I refuse to open ports for it. It works super well and is fairly easy to setup.