Selfhosted

40187 readers

687 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago

MODERATORS

How are you keeping on top of fleet updates? (reddrefuge.com)

submitted 1 year ago by [email protected] to c/selfhosted

26 comments fedilink hide all child comments

Just wondering what tools and techniques people are using to keep on top of updates, particularly security-related updates, for their self-hosting fleet.

I'm not talking about docker containers - that's relatively easy. I have Watchtower pull (not update) latest images once per week. My Saturday mornings are usually spent combing through Portainer and hitting the recreate button for those containers with updated images. After checking the service is good, I manually delete the old images.

But, I don't have a centralised, automated solution for all my Linux hosts. I have a few RasPis and a bunch of LXCs on a pair of Proxmox nodes, all running their respective variation of Debian.

Not a lot of this stuff is exposed direct to the internet - less than a handful of services, with the rest only accessible over Wireguard. I'm also running OPNsense with IPS enabled, so this problem isn't exactly keeping me up at night right now. But, as we all know, security is about layers.

Some time ago, on one of my RasPis, I did setup Unattended Upgrades and it works OK, but there was a little bit of work involved in getting it setup just right. I don't relish the idea of doing that another 40 or so times for the rest of my fleet.

I also don't want all of those hosts grabbing updates at around the same time, smashing my internet link (yes, I could randomise the cron job within a time range, but I'd rather not have to).

I have a fledgling Ansible setup that I'm just starting to wrap my head around. Is that the answer? Is there something better?

Would love to hear how others are dealing with this.

Cheers!

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago) (1 children)

I'm in the process of migrating my servers to NixOS. It takes a lot of time and the learning curve is steep, but I have one config shared for all the servers and PCs. I have setup the servers to automatically pull the latest configuration everyday and even restart if there's a kernel update.

This means I just need to update my laptop and push the changes to the repository, and all the servers will also update.

I haven't had this setup long enough to know if things will break unexpectedly with updates tho. NixOS has a great feature where you can rollback to a previous configuration (generation) with a single command. You can always keep using containers to isolate updates, if you want (Nix allows you to declare those in the config as well).

As an example, you can take a look at my config.

EDIT: Systemd timers have an option to randomize the time a service runs, I use it all the time. The option for Nix's config pulling is using systemd timers, so you can use that.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

OK, that does sound really good. Reminds me of a CVS & Perl based config management system I worked on many (many) years ago (was invented by one of the other sys admins before I got there). That was for OpenBSD, but similar concept - centralised config, pushed to client, with automation of service/server restarts as required.

I might have to consider NixOS for a long-term strategy. Cheers.