OK, so for anyone who might stumble across this in the future; I solved it. Kinda.
Basically, what's happening is that lemmy is using the site URI for the human-readable content and /api for api stuff (including login, loading and a bunch of other stuff).
I tried setting up two sites in the tunnel; one to lemmy.mydomain and one to lemmy.mydomain/api but that didn't seem to work. Presumably due to websocket calls not being re-routed.
What I opted to do was to setup lemmy.mydomain to my Nginx Reverse Proxy (I manage it using Nginx Proxy Manager). From there I added a proxy host pointing to my lemmy ui docker container and created a custom location for /api, pointing to the backend at port 8536.
The result is working great and all functions (that I've tested so far) is working without a hitch! Certificates are automatically managed by Cloudflare and I also get the adde dbenefit that Cloudflare offers on DNS and filtering while allowing access to my lemmy instance.