Update: I had a limit for my memory usage set to some default of 4 GiB I increased that to a higher number. This improved but did not resolve the issue in the end.
Anyone reading this here is the guide I followed to do so on a Debian: https://discourse.ardour.org/t/your-system-has-a-limit-to-locked-memory-solution/86813