this post was submitted on 06 Aug 2024
91 points (97.9% liked)

Selfhosted

40356 readers
361 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
ruud
Loki
CannaVet
devve
HybridSarcasm
[email protected]
91
This happens after 3-4 days of running the server, then I have to restart it manually. (discuss.tchncs.de)
submitted 3 months ago* (last edited 3 months ago) by [email protected] to c/selfhosted
38 comments fedilink hide all child comments
 

I bought an Optiplex 5040, with an i5-6500TE, and 8 GB DDR3L RAM.

When I bought it, I installed Fedora Server on it. It got stuck every few days but I could never see the error. The services just stopped working, I couldn't ssh into it, and connecting it to a monitor showed a black screen.

So, I thought let's install Ubuntu Server, maybe Fedora isn't compatible with all of its hardware. The same thing is happening, now, but I can see this error. Even when there's nothing installed on it, no containers, nothing other than base packages, this happens.

I have updated the bios. I have tried setting nouveau.modeset=0 in the grub config file. I have tried disabling and enabling c-states. No luck till now.

Would really appreciate if anyone helps me with this.

UPDATE:

  • I cleaned everything and reapplied the thermal paste. I did not see any change in the thermals. It never goes over 55°C even under full load.
  • I reset the motherboard by removing that jumper thing.
  • I ran memtest86, which took over 2½ hours. It did not show any errors.
  • I ran a CPU stress test for over 15 hours, and nothing crashed.
  • I also ran the Dell's diagnostic tool, available in the boot menu of the motherboard. The whole test took over 2 hours but did not show any errors. It tested the memory, CPU, fans, storage drives, etc.
top 38 comments
sorted by: hot top controversial new old
[–] just_another_person 53 points 3 months ago* (last edited 3 months ago) (3 children)

Seen this before, and almost always has to do with hardware failure or bad hardware config.

Reset the BIOS/CMOS jumper on the board, go back into BIOS setup and set the proper time. Do not touch the CPU or Memory timings. Boot with the defaults and see if it still happens. Check and update the BIOS if there is a newer version as well.

Next longer steps: test memory, then stress test the CPU. I'd be shocked if it was a storage issue as I haven't seen that be the culprit, but might was well run the long SMART tests.

[–] seaQueue 20 points 3 months ago (1 children)

Make sure the microcode package is up to date as well

[–] [email protected] 8 points 3 months ago

Yeah, always check all of this stuff. Server hardware gets a lot more updates than like gamer board BIOS, companies invest high millions, even low billions in this stuff and they expect problems to be address promptly for that kind of cash.

Check for any peripherals or cards, too. RAID, backplanes, networking cards; drivers, firmware, anything.

[–] [email protected] 5 points 3 months ago (1 children)

Poorly supported hardware will also do this

[–] seaQueue 8 points 3 months ago

An Optiplex 5040 should be well and thoroughly supported for 6+y now

[–] [email protected] 1 points 3 months ago

I did reset it. It did not help. I ran memtest86 for over 2 hours and did a CPU stress test for over 15 hours. Nothing crashed during the testing.

[–] [email protected] 24 points 3 months ago (1 children)

Had the same issues, it was heat.

Cool down your server, add a fan or a cooler...

I added a usb-powered fan sucking cooler air from outside the server area directly blowing it on the chassis.

That fixed for me.

[–] [email protected] 1 points 3 months ago

I cleaned everything and reapplied the thermal paste. That did not solve the problem. Also, the CPU is only of 35 watts and never goes over 55°C.

[–] [email protected] 15 points 3 months ago (1 children)

That server sounds a bit older in the teeth... Has new thermal paste been applied to the cpu? Even if the reported temps are under 90c, you might be getting hot spots causing glitches inside the package.

Worth trying a couple of different generations of kernel as well, both newer and older. You might be hitting a regression somewhere.

[–] [email protected] 2 points 3 months ago

I cleaned everything and reapplied the thermal paste. That did not solve the problem. Also, the CPU is only of 35 watts and never goes over 55°C.

[–] solrize 13 points 3 months ago (1 children)

Try running memtest86 for a few days to test memory. That is fairly easy to do though it involves booting from a flash drive. Web search should find info.

[–] [email protected] 1 points 3 months ago (1 children)

I just ran it. It took over 2 hours to finish. Showed no errors. Is there a benefit of running it for a few days?

[–] solrize 2 points 3 months ago

If the problem is intermittent then longer run has better chance of catching it, but 2 hours with no errors is a good sign, with regard to the memory.

[–] nialv7 12 points 3 months ago

Time to add a cron job to auto reboot it once a day

[–] ikidd 9 points 3 months ago* (last edited 3 months ago)

Can you sudo dmesg | grep microcode and see if you have any errors?

If so, I'd be inclined to sudo dnf reinstall microcode_ctl then do a sudo dracut -f to regenerate the initramfs, and reboot. Be sure to have a working fallback kernel like LTS installed so you can recover if need be.

Edit: I just read you changed to Ubuntu. I can't be arsed to figure out how Ubuntu does this stuff so that's on you to figure out. Alternatively, install Arch and use intel-ucode package.

[–] bulwark 7 points 3 months ago (1 children)

I've never seen this particular error, but CPU stall warnings seem like a fairly common thing. I wouldn't jump straight to hardware fault, but it's a possibility.

https://docs.kernel.org/RCU/stallwarn.html

[–] [email protected] 9 points 3 months ago (1 children)

I'd lean toward bad hardware.

Try stress testing the CPU and RAM. See if you can get it to happen more frequently. Also see if you can disable that CPU core, either in the BIOS or in the OS, to see if the problem goes away.

[–] loganb 8 points 3 months ago (1 children)

I'm with catloaf. Consistent CPU soft locks point to a possible bad memory module or CPU.

Clear CMOS.

Try removing one memory module at a time.

See if there is an option to disable hyperthreading in bios.

Another thing to try is to remove the CPU, careful not to damage the LGA pins on the motherboard, and clean the CPU contacts with alcohol. Take care to ground yourself out and the case before handling the CPU out of socket.

[–] [email protected] 3 points 3 months ago (2 children)

Don't try to clean CPU pins. That is a very bad idea

[–] loganb 4 points 3 months ago (1 children)

I mean speaking from experience, its resurrected a couple problematic CPUs for me. CPU pins no, pads on an LGA style CPU, sure.

[–] [email protected] 1 points 3 months ago

It is much harder to mess up pads

[–] [email protected] 2 points 3 months ago (1 children)

The CPU in this has no pins, is just contacts on the chip. The pins are in the motherboard, like the new 7000 series Ryzen.

[–] [email protected] 1 points 3 months ago

Clean pads not pins

[–] [email protected] 6 points 3 months ago* (last edited 3 months ago)

I had problems with soft locks because somehow the PSU was in corrupt state, maybe through a black out or something. The problem persisted through reboots and power offs, only cutting power helped.

[–] rookbrood 6 points 3 months ago (1 children)

It's a long shot, but I had something similar on one of mine servers once. It was fixed by installing irqbalance and starting that daemon at startup.

[–] [email protected] 2 points 1 month ago (1 children)

This actually worked. The CPU has to get stuck, it will in a day of being turned on, or it will keep working for weeks.

Thanks a lot for this!

[–] rookbrood 2 points 1 month ago

Haha completely forgot about this comment. But i'm very gald it helped! I remember struggling with that problem for a long time.

[–] [email protected] 5 points 3 months ago* (last edited 3 months ago)

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
LTS Long Term Support software version
PSU Power Supply Unit
RAID Redundant Array of Independent Disks for mass storage
SATA Serial AT Attachment interface for mass storage

4 acronyms in this thread; the most compressed thread commented on today has 8 acronyms.

[Thread #907 for this sub, first seen 6th Aug 2024, 10:15] [FAQ] [Full list] [Contact] [Source code]

[–] thesporkeffect 4 points 3 months ago

I'd start here: https://askubuntu.com/questions/1264859/watchdog-bug-soft-lockup-cpu6-stuck-for-23s

[–] [email protected] 3 points 3 months ago

Does it have a Nvidia GPU? If it doesn't then the nouveau modest does nothing.

My guess it that you are using a badly support realtek device. However I would need to see the full dmesg

[–] [email protected] 1 points 3 months ago (1 children)

What kind of drives do you have in your RAID? Is it SMR?

[–] [email protected] 1 points 3 months ago (1 children)

The boot drive is an SSD, which is not in any RAID. I have another HDD connected via SATA. Another HDD connected via USB.

[–] [email protected] 1 points 3 months ago* (last edited 3 months ago) (1 children)

So 2 HDDs one SATA and one via USB in RAID? Can you remove the RAID drives and test it out?

Also what size are the drives and what's their capacity?

[–] [email protected] 1 points 3 months ago (1 children)

No, they are not in RAID either.

[–] [email protected] 1 points 3 months ago (1 children)

My bad. What are you using the drives for?

[–] [email protected] 1 points 3 months ago (1 children)

Films and shows, via Jellyfin.

[–] [email protected] 1 points 3 months ago

Is it just you, any transcoding? Have you tried it without the USB drive?

[–] TheBigBrother -3 points 3 months ago* (last edited 3 months ago)

Set watchdog to reboot every day and juice it until the last drop before it definetly crashes.

Edit: that's just a workaround if you really want try to fix it my answer would be restoring BIOS defaults, then clean install of the OS and then check it everything works fine, if the error persist try installing another OS if it still fail then go to the first step.