Selfhosted

If you're willing to use Caddy instead, it's infinitely easier. Websockets is just enabled by default, no shenanigans. Here is an example Caddyfile to use in a Docker deployment (but you can change those http urls to point to localhost for a non Docker deployment):

(caddy-common) {
    encode gzip
    header {
        -Server
        Strict-Transport-Security "max-age=31536000; include-subdomains;"
        X-XSS-Protection "1; mode=block"
        X-Frame-Options "DENY"
        X-Content-Type-Options nosniff
        Referrer-Policy  no-referrer-when-downgrade
        X-Robots-Tag "none"
    }
}

your.lemmy.url {
    import caddy-common
    reverse_proxy   http://lemmy-ui:1234

    @lemmy {
        path    /api/*
        path    /pictrs/*
        path    /feeds/*
        path    /nodeinfo/*
        path    /.well-known/*
    }

    @lemmy-hdr {
        header Accept application/*
    }

    handle @lemmy {
        reverse_proxy   http://lemmy:8536
    }

    handle @lemmy-hdr {
        reverse_proxy   http://lemmy:8536
    }

    @lemmy-post {
        method POST
    }

    handle @lemmy-post {
        reverse_proxy   http://lemmy:8536
    }
}

Caddy has some great plugins that allow you to automate https certificate renewal too, easy to add to any config.

I know you asked about nginx and I'm just telling you "haha just switch," but I had similar headaches with my own deployment when I tried using nginx, and I eventually just gave up and used Caddy. Saved me at least a few hours of headache.