this post was submitted on 04 Aug 2024
132 points (94.6% liked)

Selfhosted

40188 readers
675 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
ruud
Loki
CannaVet
devve
HybridSarcasm
[email protected]
132
My homelab had the stupidest outage ever (blog.thefossrant.com)
submitted 3 months ago by [email protected] to c/selfhosted
51 comments fedilink hide all child comments
 

This morning I woke up to my phone using mobile data and my home assistant automations not working. Initially I thought it the power was out, but I could turn on the lights just fine. I checked my UniFi app and saw that the server was not connected to the network at all. This meant that the cable got unplugged, the switch isn't working, or the server isn't working. It said the switch was connected and another device was connected to the switch so that narrows it down to just 2 cases. So I opened my server closet in the basement and immediately noticed something was wrong. I couldn't figure out what was wrong but I just felt like something was wrong. Everything was plugged in, the network switch lights were blinking like normal, my raspberry pi was running just fine, even the server indicator lights were on. My main server is an old gaming PC so it has a glass side panel so I looked inside and I could see the fan spinning, but I could not hear it. Usually I have it set to full speed and I can hear full speed very well. I tried rebooting the server with the power button and the fans didn't go to full speed. As a last resort, I brought down a keyboard and monitor. As soon as I plugged in the monitor, I saw that there was a prompt to set the time on the BIOS! Picture of the prompt In my opinion, this was the stupidest reason for an outage.

Further investigations

I dug a little deeper and discovered that the BIOS had been reset during a power outage right before all of this happened. So far I have consulted the motherboard manual and found absolutely nothing about this. After a bit of research, I think it could have been that the CMOS battery has died. This is a really simple fix but I don't have the replacement battery right now. This means that I will have the same exact issue after the next power outage unless I replace the battery.

Preventing this in the future

From what I can see, I just need to replace the CMOS battery. But this computer has been running for over 4 years, so what is stopping this from happening again around 2028? The most effective solution is going to be preventing power outages in the first place. This can be done using a battery backup or a standby generator. Standby generators will last longer during a power outage but are typically more expensive and harder to setup than a simple battery backup.

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 158 points 3 months ago* (last edited 3 months ago) (6 children)
  1. Replace CMOS battery.
  2. Get small UPS.
  3. Discover that small UPS's fail regularly, usually with cooked batteries.
  4. Add maintenance routine for UPS battery.
  5. Begin to wonder if this is really worth it when the rest of the house has no power during an outage.
  6. Get small generator.
  7. Discover that small generators also need maintenance and exercise.
  8. Decide to get a whole house battery backup a-la Tesla Powerwall topped off by solar and a dedicated generator.
  9. Spend 15 years paying this off while wondering if the payback was really worth it, because you can count on one hand the number of extended power outages in that time.
  10. In the end times a roving band of thugs comes around and kills you and strips your house of valuable technology, leaving your homelab setup behind and - sadly - without power. Your dream of unlimited availability has all been for nought.

Conclusion: just replace the CMOS battery on a yearly basis during planned system downtime.

[–] [email protected] 29 points 3 months ago (2 children)

CMOS batteries last a lot longer than a year. Unless the system has been unplugged for a long time, they should be good for several years. I'm sure there's actual data out there somewhere.

But yeah, a lot of people think "oh I'll just put a UPS on it". They don't consider that unless you get a really big UPS, they're only good for very short outages, seconds or minutes, to bridge the gap between the outage and your generator coming on (or the mains power coming back if it was just a flicker).

Also, the batteries in them need to be replaced every 3-5 years.

[–] [email protected] 10 points 3 months ago (1 children)

Yeah , it's really a little strange in OPs case, I can't really recall changing a CMOS battery in ages, like decades of computer use.

[–] [email protected] 7 points 3 months ago

I had to do it for the first time last year and I was slightly giddy from the novelty of it.

[–] [email protected] 5 points 3 months ago

I bought a small APC UPS about a year ago and am glad I did. In my area, very brief outages are somewhat common so a small UPS will work for the majority of outages.

[–] [email protected] 11 points 3 months ago

What about UPS just for CMOS battery? And a tiny diesel generator with 60 ml tank

[–] [email protected] 8 points 3 months ago (1 children)

I'm somewhere between step 7 and 8.

[–] [email protected] 3 points 3 months ago (1 children)

Currently at step 9. Waiting for the roving bands of thugs to arrive 😅

[–] [email protected] 4 points 3 months ago

I scheduled them before you, so I'll let you know once they're done here (maybe).

[–] [email protected] 6 points 3 months ago* (last edited 3 months ago) (2 children)

hey, AI defense turrets sound like a cool and totally not dangerous project. hope you have a decent GPU in your setup, good luck.

[–] [email protected] 5 points 3 months ago

You can deflect rain while you're at it too.

[–] Cadeillac 2 points 3 months ago (1 children)

It is such a huge pet peeve when people say or spell turret wrong. Don't take this as me attacking you, but as an fyi

[–] [email protected] 2 points 3 months ago (1 children)

whoops typo

[–] Cadeillac 2 points 3 months ago

No worries lol, sorry

[–] [email protected] 3 points 3 months ago

Yup. I don't have a UPS (probably should; have a good surge protector though), and replacing CMOS batteries is way easier than dealing with the rest. Thanks for the reminder, I'll go pick some up and swap it out every so often.

[–] [email protected] 3 points 3 months ago

Stop, he's already dead!

[–] Khanzarate 29 points 3 months ago (1 children)

The only real solution is to make this an extended maintenance task. The batteries are cheap so an alert every 4 years is likely sufficient to replace the battery before it dies. You could do it every 2 or 3 years instead at your discretion.

[–] [email protected] 15 points 3 months ago (1 children)

The only real solution is to make this an extended maintenance task.

This is the correct answer. No matter how reliable your power feed is you still need to reboot the server at some point for whatever reason and if CMOS battery is dead by then you'll have the very same issue and you'll need monitor and keyboard again. And even if you don't mind about the RTC on board you'll still lose the settings.

I wonder why manufacturers haven't switched over to supercapacitors or something else than a coin cell battery, but perhaps there's a valid reason for it. I think that supercaps can't hold charge as long as a coin cell, but if your board is completely cold for a year or so maybe losing bios settings isn't that big of a deal.

[–] Khanzarate 4 points 3 months ago (1 children)

Well they still have a finite life and are less replaceable than a battery. Even if it quadrupled the lifespan (which is a reasonably generous estimate given OP's 4-year duration and wikipedia telling me supercapacitors last 10-15 years), it would still eventually need to be replaced and that would generally require resoldering it.

I think a much better solution is 2 battery slots, one to be a backup battery, unused, and then when needed, an LED on the mobo can be turned on. Honestly OP could jury-rig up a similar system if he wanted to, although it'd be a bit ugly and anytime something is jury-rigged I don't really think of it as reliable.

load more comments (1 replies)
[–] [email protected] 19 points 3 months ago (2 children)

Chec out this article

[–] kolorafa 4 points 3 months ago

grep bat /proc/driver/rtc ❤️

Always learning something new, thanks :)

[–] subtext 4 points 3 months ago

Man, some people have really thought of everything. I am so impressed.

[–] [email protected] 11 points 3 months ago

If you're having frequent power outages, you might consider buying an UPS. Other than that, I'd just buy a new CMOS battery every few years. Mine seem to last way longer than 4 years. Maybe 8y or so.

[–] [email protected] 9 points 3 months ago

From what I can see, I just need to replace the CMOS battery. But this computer has been running for over 4 years, so what is stopping this from happening again around 2028?

Replace the CMOS battery again.

[–] [email protected] 9 points 3 months ago (1 children)

What about all those fancy new laptops that are built like smartphone that can never be opened do they have CMOS batteries? Ive never thought if this till now.

[–] [email protected] 11 points 3 months ago (1 children)

If the main battery isn't "meant to be replaced", it will often act as the CMOS battery (e.g. MacBooks have been doing this since roughly 2008).

[–] [email protected] 4 points 3 months ago

Huh cool

[–] [email protected] 7 points 3 months ago (1 children)

Why you are at it you might want to check the battery in your smoke alarm. I think your are support to change the smoke alarm battery every 5 years.

[–] [email protected] 6 points 3 months ago* (last edited 3 months ago) (4 children)

The battery in smoke detectors should be changed every year and you should ideally test the detectors using the little button monthly.

In Norway we have an official smoke detector day which is on December 1. where we're supposed to change the batteries. Its because Christmas is the high season for house fires.

[–] [email protected] 4 points 3 months ago (1 children)

That depends on the type of battery. I have some of the old school ones, which are "test every 6 months, replace every year," but also a fancy schmancy lithium ion-powered one that should be good for 10 years (so test every 6 months, but replace after 10 years).

[–] [email protected] 4 points 3 months ago

Yea the lithium-ion batteries are pretty good, you can also get 10 year smoke detectors with a built in lithium battery. Also a psa, smoke detectors usually only last 10 years and need to be replaced after that. There's often an expiry date on the detectors.

load more comments (3 replies)
[–] [email protected] 6 points 3 months ago (1 children)

If you're the owner of the website, I suggest you look into those php errors


Warning: simplexml_load_file(https://feeds.thefossrant.com/rss_feed.xml): failed to open stream: php_network_getaddresses: getaddrinfo failed: No address associated with hostname in /var/www/html/article.php on line 2

Warning: simplexml_load_file(): I/O warning : failed to load external entity "https://feeds.thefossrant.com/rss_feed.xml" in /var/www/html/article.php on line 2

Warning: Invalid argument supplied for foreach() in /var/www/html/article.php on line 5

Warning: Invalid argument supplied for foreach() in /var/www/html/article.php on line 19

Published on: 1970-01-01 00:00:00 UTC

Tags:
Image unavailable
[–] [email protected] 7 points 3 months ago (1 children)

I am aware. There are some issues with the DNS

[–] Opisek 7 points 3 months ago

It's always the DNS!

[–] [email protected] 6 points 3 months ago

I just do an uptime -p if I suspect there was a power outtage

[–] EarMaster 5 points 3 months ago (1 children)

Some BIOS manufacturers allow you to disable all halts on errors. As soon as you connect to the network your system should be able to sync its internal clock.

[–] [email protected] 7 points 3 months ago (1 children)

Some BIOS manufacturers allow you to disable all halts on errors.

That will be reset to default if the CMOS battery is dead and power is removed though.

[–] computergeek125 3 points 3 months ago (1 children)

Not on a flash based motherboard (so basically almost everything recent). On modern systems usually the only thing the battery powers is the clock, which is why they have a separate reset to defaults header/button/switch.

(The CMOS memory of old is replaced with flash memory, al la SD Card or flash drive)

[–] [email protected] 2 points 3 months ago

Gotcha, that's handy.

[–] TCB13 5 points 3 months ago

Here's the thing, you now know you've to replace the battery every 3 years or so, but I've SONY batteries running for 5 years and they're fine. Now, your motherboard's manufacturer is a piece of shit, the board should've just ignored the issue and proceeded to the OS. All modern operating systems sync the UEFI time whenever they get time from NTP so no need to halt the boot process.

[–] [email protected] 3 points 3 months ago

Honestly just replace the CMOS battery on a schedule if it's a big deal, a UPS is nice to have but it doesn't really solve that specific problem.

[–] [email protected] 2 points 3 months ago

I agree with you about an uninterruptible power supply. While a computer’s Power supply can normally handle about half a second of the wrong voltage, a UPS should provide you with minutes to hours of run time. While that won’t solve the CMOS going bad, it would probably reduce the amount of times a power outage would last long enough to shut down the computer - though that absolutely depends on the reliability of power in your area.

Another option would be to look at different motherboard options. AsRock Rack (and I believe Supermicro) offer boards with BMCs, so you’d be able to change your BIOS settings from anywhere with a virtual KVM. They’re certainly pricier options, but a lovely convenience to have.

[–] [email protected] 2 points 3 months ago (1 children)

Can you replace the CMOS battery with a supercapacitor that is kept charged? This should not need to be replaced every 4 years, I think.

[–] [email protected] 2 points 3 months ago

We use these in our products at work and I’ve never heard about them failing before the equipment is replaced anyways. They are soldered.

I won’t quote life expectancy (MTBF) but we have many customers with 15-20 year old PLC’s with the original capacitor.

[–] [email protected] 2 points 3 months ago

I wonder.... How about a hot swap cmos battery? Thinking you wire the cmos terminals to a few batteries in parallel Or 3 And add a voltage indicator for each of them

load more comments
view more: next ›