Hello there friendly users!
Update 3:50pm CEST
The container has been moved, most work on the new host has been completed. There are some permanent data errors on the original storage pool, but it does not like it affected this linux container. Please give DNS and federation a moment.
Some (but only a few) of you have experienced an outage today around 5-8am CEST. This – as far as my current findings go – happened due to a filesystem panic. So far, i only experienced such a thing on virtual servers with a certain amount of load.
Even tho relatively speaking, i didn't feel like there was much to worry about with the current setup and its utilization, i feel like when looking at a potential second wave from Reddit, this instance should be prepared – so i just did a thing:
What we are moving to
We will be moving from a Netcup RS 4000 G9.5 KVM with 32 GB memory and 10 dedicated EPYC cores, (that was meant for various smaller tchncs services such as XMPP, Email, Clouds, Websites and stuff and lots of headroom), to a Hetzner serverauction baremetal thing with an Intel Core i7-7700, 2x SSD M.2 NVMe 512 GB, 64 GB DDR4 Mem, 1 Gbit Networking in Finland.
I plan to migrate today as soon as the thing is ready and configured. There will be a downtime of up to 1 hour.
Lemmy will share its seat on the machine with the tiny Calckey instance that was deployed around the same time.
Increased monthly cost: around 40 EUR
Note on the thing Meta is developing with ActivityPub federation support:
I am not a friend of Meta and am happy to keep it out of my life an off of my servers. However there are lots of rumors and barely any confirmed informations around the project. I will act accordingly once more is known. I also won't make any exceptions in terms of enforcing our rules. However i won't participate in signature lists for pre-blocking everything Meta and run around in circles, screaming in panic.
Is there a metrics/grafana page where I can see the bandwidth and compute that the server uses? Do you have insights into that? Where do you see the bottlenecks in the current system?
Not right now, sorry, but i might add something like this (again) in the near future. There was some memory leaky behavior and a crying ZFS of this kind never happened to me on my dedicated hosts so far. Otherwise the previous system was fine... hard to predict the next incoming wave and stuff tho.
Np. Don't worry about it, I don't want to hassle you into doing extra (potentially) unnecessary work. Interesting that ZFS was leaking - I have no experience with it, I just heard that ZFS in general requires a lot of RAM to be a reasonable filesystem choice. Are you sure it was a leak, or could it have been e.g. caches filling up as well?