this post was submitted on 28 Jun 2023

1867 points (99.1% liked)

Lemmy.World Announcements

29096 readers

340 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

DM https://lemmy.world/u/lwreport
Email [email protected] (PGP Supported)

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Join the team

founded 2 years ago

MODERATORS

1867

[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back (self.lemmyworld)

submitted 1 year ago* (last edited 1 year ago) by ruud to c/lemmyworld

351 comments fedilink hide all child comments

We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

you are viewing a single comment's thread
view the rest of the comments

[–] snargledorf 14 points 1 year ago (2 children)

@[email protected] just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?

With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.

[–] ruud 13 points 1 year ago (2 children)

Yes, considering that. But we'll need people to use that when we will do testing...

[–] snargledorf 10 points 1 year ago (1 children)

Well, you have at least one person who would be willing to use it ;D

[–] [email protected] 7 points 1 year ago

One more. You find a stable way to notify anout upgrades and get a test sheet to run through and we can generate posts and activity to help test with.

Light the beacons! lemmy.world calls for aid!

[–] Finnagain 3 points 1 year ago (1 children)

I'm not familiar with what the server architecture looks like, but is there a possibility of using a load balancer in front of the instance's server and swapping a "beta" server into the load balancer when you need to do testing? You could basically migrate your traffic with zero downtime, assuming Lemmy's architecture allows for it.

[–] ruud 2 points 1 year ago (1 children)

Well that doesn't differ much from what I do. I just copy the files to a second directory and test with that. Easy rollback. Downside is, that all data is lost between upgrade and rollback, which will be the same in the scenario you suggest.

[–] Finnagain 2 points 1 year ago

Ah, that's fair. Best of luck either way. This is the tough part of admin work. FWIW, the stability we've had thus far has been pretty impressive!

[–] neblem 2 points 1 year ago (1 children)

Is there a way to have two instances writing to the same db? That'd help a lot in this situation so the test instance can still be accessing the same data as the stable instance. Otherwise we'll never have enough load to fully stress test and be in the same situation as the three existing Lemmy test instances.

[–] snargledorf 6 points 1 year ago

Seeing that with these updates often come with changes to the database as well, that could be very difficult if not impossible.