this post was submitted on 24 Apr 2024
110 points (99.1% liked)

Selfhosted

40397 readers
647 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

I recognize this will vary depending on how much you self-host, so I'm curious about the range of experiences from the few self-hosted things to the many self-hosted things.

Also how might you compare it to other maintenance of your other online systems (e.g. personal computer/phone/etc.)?

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 76 points 7 months ago* (last edited 7 months ago) (6 children)

Huge amounts of daily maintenance because I lack self control and keep changing things that were previously working.

[–] [email protected] 20 points 7 months ago (2 children)

highly recommend doing infrastructure-as-code, it makes it really easy to git commit and save a previously working state, so you can backtrack when something goes wrong

[–] kernelle 7 points 7 months ago

Ansible is great for this!

[–] [email protected] 6 points 7 months ago (1 children)

Got any decent guides on how to do it? I guess a docker compose file can do most of the work there, not sure about volume backups and other dependencies in the OS.

[–] kernelle 2 points 7 months ago (1 children)

Sorry I replied to the parent comment, but check out Ansible

[–] [email protected] 3 points 7 months ago (1 children)

Oh I think i tried at one point and when the guide started talking about inventory, playbooks and hosts in the first step it broke me a little xd

[–] kernelle 3 points 7 months ago* (last edited 7 months ago)

I get it, the inventory is just a list of all servers and PC you are trying to manage and the playbooks contain every step you would take if you would configure everything manually.

I'll be honest when you first set it up it's daunting but that's the thing! You only need to do it once, then you can deploy and redeploy anything you have in minutes.

Edit: found this useful resource

load more comments (5 replies)
[–] [email protected] 52 points 7 months ago (2 children)

sometimes I remember I'm self hosting things

[–] BigMikeInAustin 7 points 7 months ago (2 children)

As long as you remember before you turn off the computer!

[–] grue 11 points 7 months ago (1 children)

I don't understand. "Turn... off?"

[–] Opisek 5 points 7 months ago

neofetch proudly displaying 5 months of uptime

[–] [email protected] 3 points 7 months ago

my main PC hosts nothing, everything else is always on

[–] seaQueue 3 points 7 months ago

+1 automate your backup rolling, setup your monitoring and alerting and then ignore everything until something actually goes wrong. I touch my lab a handful of times a year when it's time for major updates, otherwise it basically runs itself.

[–] [email protected] 37 points 7 months ago* (last edited 7 months ago) (2 children)

Very minimal. Mostly just run updates every now and then and fix what breaks which is relatively rare. The Docker stacks in particular are quite painless.

Couple websites, Lemmy, Matrix, a whole email stack, DNS, IRC bouncer, NextCloud, WireGuard, Jitsi, a Minecraft server and I believe that's about it?

I'm a DevOps engineer at work, managing 2k+ VMs that I can more than keep up with. I'd say it varies more with experience and how it's set up than how much you manage. When you use Ansible and Terraform and Kubernetes, the count of servers and services isn't really important. One, five, ten, a thousand servers, it matters very little since you just run Ansible on them and 5 minutes later it's all up and running. I don't use that for my own servers out of laziness but still, I set most of that stuff 10 years ago and it's still happily humming along just fine.

[–] [email protected] 4 points 7 months ago

+1 for docker and minimal maintenance. Only updates or new containers might break stuff. If you don’t touch it, it will be fine. Of course there might be some container specific problems. Depends what you want to run. And I’m not a devops engineer like Max 😅

load more comments (1 replies)
[–] 0110010001100010 12 points 7 months ago

Typically, very little. I have ~40 containers in my Docker stack and by in large it just works. I upgrade stuff here and there as needed. I am getting ready to do a hardware refresh but again with Docker that's pretty painless.

Most of the time spent in my lab is trying out new things. I'll find a new something that looks cool and go down the rabbit hole with it for a while. Then back to the status quo.

[–] [email protected] 11 points 7 months ago (2 children)

Once setup correctly, almost none.

load more comments (2 replies)
[–] CarbonatedPastaSauce 11 points 7 months ago

It's bursty; I tend to do a lot of work on stuff when I do a hardware upgrade, but otherwise it's set it and forget it for the most part. The only servers I pay any significant attention to in terms of frequent maintenance and security checks are the MTAs in the DMZ for my email. Nothing else is exposed to the internet for inbound traffic except a game server VM that's segregated (credential-wise and network-wise) from everything else, so if it does get compromised it would be a very minimal danger to the rest of my network. Everything either has automated updates, or for servers I want more control over I manually update them when the mood strikes me or a big vulnerability that affects my software hits the news.

TL;DR If you averaged it over a year, I maybe spend 30-60 minutes a week on self hosting maintenance tasks for 4 physical servers and about 20 VM's.

[–] dlundh 8 points 7 months ago

A lot less since I started using docker instead of running separate vms for everything. Less systems to update is bliss.

[–] Opisek 7 points 7 months ago

As others said, the initial setup may consume some time, but once it's running, it just works. I dockerize almost everything and have automatic backups set up.

[–] [email protected] 7 points 7 months ago

For some reason my DNS tends to break the most. I have to reinstall my Pi-hole semi-regularly.

NixOS plus Docker is my preferred setup for hosting applications. Sometime it is a pain to get running but once it does it tends to run. If a container doesn't work, restart it. If the OS doesn't work, roll it back.

[–] [email protected] 6 points 7 months ago

It's as much or as little as you want to. If you don't want to change anything, you can use something like debian and only maintain once every 5 years (and you could even skip that).

I personally spend a little more, by choice, because I use gentoo. But if I'm busy, I can avoid maintenance by only running routine updates every couple of weeks or so.

[–] [email protected] 5 points 7 months ago

I spend a huge amount of time configuring and setting up stuff as it's my biggest hobby. But I got good enough that when I set something up it can stay for months without any mainainence. Most I do for keeping it up is adding more storage if it turn out to be used more than planned.

[–] [email protected] 5 points 7 months ago* (last edited 7 months ago)

Not heaps, although I should probably do more than I do. Generally speaking, on Saturday mornings:

  • Between 2am-4am, Watchtower on all my docker hosts pulls updated images for my containers, and notifies me via Slack then, over coffee when I get up:
    • For containers I don't care about, Watchtower auto-updates them as well, at which point I simply check the service is running and purge the old images
    • For mission-critical containers (Pi-hole, Home Assistant, etc), I manually update the containers and verify functionality, before purging old images
  • I then check for updates on my OPNsense firewall, and do a controlled update if required (needs me to jump onto a specific wireless SSID to be able to do so)
  • Finally, my two internet-facing hosts (Nginx reverse proxy and Wireguard VPN server) auto-update their OS and packages using unattended-upgrades, so I test inbound functionality on those

What I still want to do is develop some Ansible playbooks to deploy unattended-upgrades across my fleet (~40ish Debian/docker LXCs). I fear I have some tech debt growing on those hosts, but have fallen into the convenient trap of knowing my internet-facing gear is the always up to date, and I can be lazy about the rest.

[–] [email protected] 5 points 7 months ago

As a complete noob trying to make A TrueNAS server, none and then suddenly lots when idk how to fix something that broke

[–] [email protected] 5 points 7 months ago* (last edited 7 months ago)

Minimal, I have to force myself to check the servers for updates atleast once a week.

Main problem for me is I automated podman and docker updates with their respective autoupdate mechanisms and use ntfy for push notifications so I know if a service stops working and I had an update recently on it that it's an update issue.

Also have uptime monitor wih uptime kuma to monitor state of my services to catch them not working before I do, also ntfy for push notifications.

Also have grafana+prometheus seted up on my biggest server for monitoring and alerting with alertmanager+mail to get notifications on even more errors.

So in general I only have to worry about occasional once every few months error and updates of the host system (debian).

[–] Crogdor 5 points 7 months ago

Mostly nothing, except for Home Assistant, which seems to shit the bed every few months. My other services are Docker containers or Proxmox LXCs that just work.

[–] hperrin 4 points 7 months ago (1 children)

If you set it up really well, you’ll probably only need to invest maybe an hour or so every week or two. But it also depends on what kind of maintenance you mean. I spend a lot of time downloading things and putting them in the right place so that my TV is properly entertaining. Is that maintenance? As for updating things, I’ve set up most of that to be automatic. The stuff that’s not automatic, like pulling new docker images, I do every couple weeks. Sometimes that involves running update scripts or changing configs. Usually it’s just a couple commands.

load more comments (1 replies)
[–] EncryptKeeper 4 points 7 months ago (1 children)

If you’re not publicly exposing things? I can go months without touching it. Then go through and update everything in an hour or so on the weekend.

[–] [email protected] 4 points 7 months ago

Too much, just, too much

[–] [email protected] 4 points 7 months ago* (last edited 7 months ago)

Maybe 1 hr every month or two to update things.

Thinks like my opnsense router are best updated when no one else is using the network.

The docker containers I like to update manually after checking the release logs. Doesn't take long and I often find out about cool new features perusing the release notes.

Projects will sometimes have major updates that break things and I strongly prefer having everything super stable until I have time to sit down and update.

11 stacks, 30+ containers. Borg backups runs automatically to various repositories. Zfs auto snap snot also runs automatically to create rapid backups.

I use unraid as a nas and proxmox for dockers and VMs.

[–] thirdBreakfast 4 points 7 months ago* (last edited 7 months ago) (1 children)

I run two local physical servers, one production and one dev (and a third prod2 kept in case of a prod1 failure), and two remote production/backup servers all running Proxmox, and two VPSs. Most apps are dockerised inside LXC containers (on Proxmox) or just docker on Ubuntu (VPSs). Each of the three locations runs a Synology NAS in addition to the server.

Backups run automatically, and I manually run apt updates on everything each weekend with a single ansible playbook. Every host runs a little golang program that exposes the memory and disk use percent as a JSON endpoint, and I use two instances of Uptime Kuma (one local, and one on fly.io) to monitor all of those with keywords.

So -

  • weekly: 10 minutes to run the update playbook, and I usually ssh into the VPS's, have a look at the Fail2Ban stats and reboot them if needed. I also look at each of the Proxmox GUIs to check the backs have been working as expected.
  • Monthly: stop the local prod machine and switch to the prod2 machine (from backups) for a few days. Probably 30 minutes each way, most of it waiting for backups.
  • From time to time (if I hear of a security update), but generally every three months: Look through my container versions and see if I want to update them. They're on docker compose so the steps are just backup the LXC, docker down, pull, up - probs 5 minutes per container.
  • Yearly: consider if I need to do operating systems - eg to Proxmox 8, or a new Debian or Ubuntu LTS
  • Yearly: visit the remotes and have a proper check/clean up/updates
load more comments (1 replies)
[–] [email protected] 3 points 7 months ago

Not much for myself, like many others. But my backups are manual. I have an external drive I backup to and unplug as I intentionally want to keep it completely isolated from the network in case of a breach. Because of that, maybe 10 minutes a week? Running gentoo with tons of scripts and docker containers that I have automatically updating. The only time I need to intervene the updates is when my script sends me a push notification of an eselect news item (like a major upcoming update) or kernel update.

I also use a custom monitoring software I wrote that ties into a MySQL db that's connected to with grafana for general software, network alerts (new devices connecting to network, suspicious DNS requests, suspicious ports, suspicious countries being reached out to like china, etc) or hardware failures (like a raid drive failing).... So yeah, automate if you know how to script or program, and you'll be pretty much worry free most of the time.

[–] [email protected] 3 points 7 months ago

If my ISP didn't constantly break my network from their side, I'd have effectively no downtime and nearly zero maintenance. I don't live on the bleeding edge and I don't do anything particularly experimental and most of my containers are as minimal as possible

I built my own x86 router with OpnSense Proxmox hypervisor Cheapo WiFi AP Thinkcentre NAS (just 1 drive, debian with Samba) Containers: Tor relay, gonic, corrade, owot, apache, backups, dns, owncast

All of this just works if I leave it alone

[–] chrundle 3 points 7 months ago

My mini-pc with Debian runs RunTipi 24/7 with Navidrome, Jellyfin and Tailscale. Once every 2-3 weeks I plug in the monitor to run updates and add/remove some media.

[–] eluminx 3 points 7 months ago

Maybe 1-2 hours a week for ~23 docker containers, 3 LXCs and proxmox, so not much. Most of that time is spend SSH-ing doing minor updates. Running Debian on everything has been amazing. Stability is just phenomenal.

[–] [email protected] 3 points 7 months ago

It's very minimal in normal use, maybe like an hour or two a month at most.

[–] clavismil 3 points 7 months ago

Like 1 hour every two months or so, I just run an ansible playbook and check everything is working ok

[–] [email protected] 3 points 7 months ago* (last edited 7 months ago)

Synology user running some docker containers.

Very, very little maintenance. If there's an update for something on docker, a simple click in the container manager, and it's done. Yes, I can automate, but prefer to manually do these as many of the docker apps I use are in high development and I like to know what's changing with each version.

Synology packages update easily, and the system updates happen only once in a while. A click and reboot.

I've tried to minimize things as much as possible, and to make things easier for me. One day, someone in my family will need to take over, and I don't want to over-complicate things for them, lest they lose all our family photos, documents, etc.

I probably spend more time keeping the fans on my actual NAS clean of dust, than I do maintain the software end of things. LOL

edit: spelling

[–] [email protected] 3 points 7 months ago

I have just been round my small setup and run an OS update, took about an hour. That includes a reboot of a dedicated server with OVH.

a pi and mini PC at home, a dedi at OVH running 2 LXC and 5 qemu vms. All deb a mix of 11 and 12.

I spend Wednesday evenings checking what updates need installing, I get an email every week from newreleases.io with software updates and run Semaphore to check on OS updates.

[–] [email protected] 3 points 7 months ago (2 children)

For my local media server? Practically none. Maybe restart the system once a month if it starts getting slow. Clear the cache, etc.

When I hosted game servers: Depending on the game, you may have to fix something every few hours. Arma 3 is, by far, the worst. Which really sucks because the games can last really long, and it can be annoying to save and load with the GM tool thing.

load more comments (2 replies)
[–] Deckweiss 3 points 7 months ago* (last edited 7 months ago) (4 children)

After my Nextcloud server just killed itself from an update and I ditched that junk software, nearly zero maintenance.

I have

  • autoupdates on.
  • daily borgbackups to hetzner storage box.
  • auto snapshots of the servers and hetzer.
  • cloud-init scripts ready for any of the servers.
  • Xpipe for management
  • keepass as a backup for all the ssh keys and password

And I have never used any of those ... it just runs and keeps running.

I am selfhosting

  • a website
  • a booking service for me
  • caldav server
  • forgejo
  • opengist
  • jitsi

I need to setup some file sharing thing (Nextcloud replacement) but I am not sure what. My usecase is mainly 1) Archiving junk 2) syncing files between three devices 3) streaming my music collection

load more comments (4 replies)
[–] [email protected] 2 points 7 months ago

Very little. I have enough redundancy through regular snapshots and offsite backups that I'm confident enough to let Watchtower auto-update most of my containers once a week - the exceptions being pihole and Home Assistant. Pihole gets very few updates anyway, and I tend to skip the mid-month Home Assistant updates so that's just a once a month thing to check for breaking changes before pushing the button.

Meanwhile my servers' host OSes are stable LTS distros that require very little maintenance in and of themselves.

Ultimately I like to tinker, but once I'm done tinkering I want things to just work with very little input from me.

[–] Presi300 2 points 7 months ago* (last edited 7 months ago)

I just did a big upgrade to my "home lab" (got a new switch and moved it out of my bedroom), which required some maintenance in the days after the upgrade... Running a new ethernet cable, because the old one just couldn't heck doing gigabit, reconfiguring my router and AP, just general stuff like that.

Other than that and my DHCP/DNS VM sometimes forgetting to autostart after a power outage, pretty much 0 maintenance

[–] ssdfsdf3488sd 2 points 5 months ago

Almost none now that i automated updates and a few other things with kestra and ansible. I need to figure out alerting in wazuh and then it will probably drop to none.

[–] [email protected] 2 points 7 months ago

Sometimes its real easy and I‘m taking a month off and nothing breaks. Then I have times where I want to add new services or optimize stuff. This can take forever. Right now I‘m building object storage behind a vpn.

load more comments
view more: next ›