Playing modded Witcher 3 through proton. After dying and loading the last save (and perodically while playing) gpu clocks drop such that the framerate is barely above 60 (and very stuttery). Tried enabling framegen, same thing but it drops to 70 ish. It sometimes goes back to normal if I alt tab out and back, but that gets annoying when playing with a controller (especially if it happens in the middle of combat).

I can see the gpu clock dropping and going back up in system monitor. This doesn't happen with any other game. Any advice?