Yeah, hardware encoding is very new, but you can get an Intel Arc 700-series card and get it via QuickSync for pretty cheap if you have a spare x16 slot and really, really need it right now. Only those, Nvidia's 4000 series, and AMD's 7000 series support encode. 3000 and 6000 series for each company support hardware decode, though.
Just need to get phones on the hardware decode train! Qualcomm and Apple, step up your game!
I, too, have only been using Handbrake because idk what else to use, so I've been limited to SVT-AV1 for encoding. I'd need to watch through stuff at length, but at speed 6 and quality 40 I personally don't notice anything super off in my 1080p Blu-ray rip of The Last Wish (and got it down to an incredible 1.5GB). I definitely didn't catch anything at speed 6 and quality 30 - it won't be as small, and it'll still take a while, but at that I personally saw like. no problems whatsoever.
Might need to watch out for any film grain though, idk how much that mucks with the process but there are settings to denoise and reapply on playback. I haven't experimented with them too much because a) handbrake, and b) was trying to find good settings on a movie without grain to start.
I mostly saw problems when I used AOM-AV1 for encoding, at speed 7 and 6000kbps. That was purely because real-time encoding was why I needed that and it just wasn't quite perfect for 1080p60 Splatoon clips, mostly also as a test for "if Twitch turned AV1 on tomorrow what could I get at their current limits." SVT instantly got overloaded for real time at any real-time speed for some reason.