Lemmy 0.18.4 has been released. You can see the official release announcement here. This one's a small one so should go smoothly. As usual, we'll update at 04:00 UTC tonight.

Edit: Completed successfully as expected. Unfortunately we did have a significant downtime just before the update due to the serve running out of memory. This was partially an odd coincidence and partly my fault. I had noticed that we were getting very near being out of memory, but expected the server would hold out until we rebooted during the maintenance window tonight. Unfortunately that didn't happen. Lemmy does gradually eat memory over time and requires periodic restarts, but the frequency seems rather variable. I never had restart the server for the entire time we were running on 0.18.2, while 0.18.3 needed a restart after just a handful of days of operation. Most significantly, this incident revealed that some of my monitoring tools aren't reporting correctly. I'll be working on improving that in the future. I may also institute a periodic server restart as well. As long as I only have to restart Lemmy and not the entire server, this should be quick enough that no one will really notice.

Lemmy 0.18.3 has been officially released. You can read the official release notes here. The eagle-eyed among you may notice that we are upgrading from 0.18.1 to 0.18.3 and wonder what happened to 0.18.2, and I share you confusion. It never got an official GitHub release so I never upgraded to it.

I plan to update tonight at ~~2023-07-29~~ ~~2023-07-30~~ 2023-07-31 04:00 UTC. Expected downtime is slightly longer than usual due to database updates, but should still be less than 10 minutes. As always, downtime may extend to up to an hour if any thing goes wrong.

Edit: There was a glitch with the update and I had to restore from a backup, which took most of the planned hour. I did successfully test the update on a clone of the production VM beforehand so I'm hopeful that simply retrying the upgrade will succeed, but to keep things within my one hour maintenance window I'll hold off on trying again until the same time tomorrow.

Edit 2: Sorry folks, same result attempting the update tonight. I did at least get a much better capture of the logs as I was prepared this time. I'm indefinitely postponing the update while I investigate and come up with a solution. If I can repeat the update successfully again on a test server I have some ideas on how to use that as a migration path. Will also be reaching out to other instance admins on Matrix to see if this is a know issue. Will keep you all updated with a new post when I have something new to share.

Edit 3: I had an epiphany last night as I was going to bed. I tested it this morning and my guess was correct. Basically I'm just an impatient idiot. This update requires a significant database migration, during which time the lemmy backend doesn't respond to requests. The frontend is up and running though and is able to display en error message. Being cautious about downtime, I saw the error message and defaulted to getting things to a known-good state withing the one hour maintenance window I had published. I suspect I was multi-tasking a bit while testing the upgrade so I just never got around to loading the test instance until after the database migration had completed. Will proceed with the update tonight with an improved understanding of what to expect.

Edit 4: Update completed. Everything looks good.

We are slowly accumulating both image and database data and our current disk will likely be full within the next several days. Fortunately I can increase the size of the disk by another 30GB without incurring any additional cost, which should be enough to at least another month of data. Unfortunately Linode doesn't let me increase the size while the server is online so there will be a brief downtime while I shut down the server and reallocate the disk.

Edit: Completed with just over 1.5 minutes of downtime.

Lemmy 0.18.1 has been officially released. You can read the official release notes here.

I plan to update tonight at 2023-07-07 04:00 UTC. Expected downtime is just a couple of minutes, but may extend to up to an hour if any thing goes wrong.

Edit: After a slight delay unrelated to the server, the update was uneventful and looks like it was successful.

Hi Everyone,

This is just a quick FYI that I will be upgrading the instance to the Lemmy 0.18.0 release today at 21:00 (9PM) Pacific time, AKA 2023-06-23 04:00 UTC. Expected downtime is just a few minutes, but may extend up to an hour if anything does not go to plan. You can read the full release announcement here.

Edit: As of 21:03, everything appears to have gone as planned and we are now running 0.18!

I caught this fast enough that I wonder if anyone even noticed, but this is the first unplanned downtime for the instance I'll go ahead and report on it anyway.

The instance has been consistently running at approximately 75-80% memory usage since it started. This morning (in my local time zone) the server experienced a memory spike. This spike caused the kernel to start swapping excessively, which in turn spiked CPU usage to 100%.

Timeline of Events

• 2023-06-14 17:11 UTC: Memory hit 100%, caused excessive swapping and 100% CPU usage
• 2023-06-14 17:14 UTC: Issue was noticed thanks to monitoring tools
• 2023-06-14 17:15 UTC: Shut down node, initiated node resize to double RAM and CPU
• 2023-06-14 17:17 UTC: Server back up and running

Evaluation

I was already a little concerned about running the server so close to the edge of memory. This event proves that it's not tenable. With the new node we now have 4GB of RAM and 2 CPU cores compared to 2GB and 1 core before. We are currently sitting at 36% memory usage and ~6% CPU. In general Lemmy is CPU bound for large user counts, so this server size should support us up to several hundred users. I expect memory will not be a concern for the foreseeable future.

As for what caused the event in the first place, I'm not sure. Server logs around the time of the event don't look unusual, but it's not practical to log all inbound activity since every comment and every vote on every instance that we are subscribed to is a separate HTTP request. It's possible that the spike was a result of bursty federation messages from lemmy.ml, which has been struggling under load the last couple of days.

I feel pretty confident about the new server hardware for now so I will continue to diligently monitor system performance, but I don't expect this particular issue to recur.

We're getting a slow but steady stream of signups, but most of you are just lurking. Creating at least one post or comment helps boost our active user count, which in turn gets us higher on the join-lemmy.org website so that more people see our little corner of the Fediverse when deciding where to sign up. I don't expect to ever be one of the major servers of the Lemmiverse, but I do think the more people we have here the more fun it will be.

To that end, please leave a comment here introducing yourself to the server, especially if you have been lurking up till now. Or, just go out there and comment anywhere on Lemmy (it doesn't have to be on our instance) so that you are counted as active!

We're slowly accumulating users here so I figure I will throw this out there. Right now pretty much all of the icons and banners on the instance are AI generated art courtesy of Midjourney. While that has its own appeal given my username, I would be more than happy to replace it with user-submitted improvements (even more AI generated stuff. I'm not all that great at getting what I want out of the AI anyway). In exchange I can offer you a shout out in the community/instance description or something in addition to the karmic benefit of contributing to the instance. :)

As you all know, Lemmy is a rapidly evolving platform. That goes for those of us administering instances as well as users. As we grow and learn, it is occasionally necessary to restart the Lemmy instance to tweak settings.

Generally speaking, I'm able to set these changes up first, then click the button to restart the server, so the instance is down for less than a minute at a time. I think this is short enough that it's not worth making a post and waiting for an announced time for every restart. When/if I encounter a situation where I need to make more significant changes they will definitely be announced.

I would like to hear whether anyone has noticed and/or is bothered by occasional unannounced downtimes. If there's anyone out there that wants me to announce them first, I'm happy to do so when possible. The only cost is that it slows down my ability to iterate towards the ideal server config.

I'm not above posting reddit links to interesting OC