linuxmemes

At one point, my laptop's Nvidia drivers were all tangled up. The package dependency graph had portions of the screwy, we-don't-need-your-stinking-standard-version-scheme, binary blob drivers both in front of and behind the currently installed version. I had to basically gut everything Nvidia related, by performing surgery on the filesystem and Apt database, and then build it back. At one point, I was flying in text mode only; not hard, but worth mentioning since it shows how deep a cut this was.

Related: getting the above nonsense to cooperate with containers that also want to do GPU things. As much as I wanted this work with coding up a one-and-done solution (e.g. docker-compose or BASH script), you can't get away with mounting the host Nvidia driver and tools via volumes. The software on the container image itself must be built against the specific version you're running - no exceptions. So, I now rebuild these containers after every Nvidia package upgrade (from the author's git repo), which is a stupid way to achieve containerization. If Nvidia had a stable API/ABI across releases, this would just work. /rant