ip:port <-> ip:port
From any particular host (be it on the WAN or LAN) every TCP/UDP transmission is sent from some specific address-port pair destined for some other specific address-port pair. From the WAN (i.e. the Internet), every destination address must be in a public range, and we ran out of those a while ago, which is why NAT became a thing at all.
Your router is the only machine on your LAN that also has a WAN address, so every transmission destined for inside your LAN must be (from the perspective of the Internet) addressed to some port on your router. Port numbers under 1024 are special, but most of the 60-thousand other ports are without special meaning, and these unremarkable ports are the ones used to send outgoing transmissions even if the destination is some well-known, meaningful port like 80 (HTTP) or 22 (SSH). When the server responds (such as with an HTTP GET result) they send the response to the address-port pair that sent the originating request.
The magic ingredient in NAT is that your router remembers that it just proxied a request from some LAN station, and it holds in reserve whichever port it used to send that request (since it knows that any responses from the WAN will be aimed at that port of the router).
When your router receives a transmission from the WAN, it consults the records it has kept to decide which LAN station is supposed to received that transmission. Here we get to the concept of Port Forwarding, which just short circuits that NAT lookup and assigns some arbitrary port on the router as a persistant pathway to some specific LAN station.
In short, yes, only the destination port is required for your router to decide.