Lemmy Server Performance

420 readers

1 users here now

lemmy_server uses the Diesel ORM that automatically generates SQL statements. There are serious performance problems in June and July 2023 preventing Lemmy from scaling. Topics include caching, PostgreSQL extensions for troubleshooting, Client/Server Code/SQL Data/server operator apps/sever operator API (performance and storage monitoring), etc.

founded 1 year ago

MODERATORS

[email protected]

De-facto memory leak (self.lemmyperformance)

submitted 1 year ago by DreadTowel to c/[email protected]

9 comments fedilink hide all child comments

It looks like the lack of persistent storage for the federated activity queue is leading to instances running out of memory in a matter of hours. See my comment for more details.

Furthermore, this leads to data loss, since there is no other consistency mechanism. I think it might be a high priority issue, taking into account the current momentum behind growth of Lemmy...

you are viewing a single comment's thread
view the rest of the comments

[–] phiresky 4 points 1 year ago (1 children)

a large part of the queue going unbounded is due to the retry queue and missing checks if the receiving servers are actually available. quick fix is disabling the retry queue which is currently making it not go unbounded on lemmy.world

storing the queue persistently is somewhat of a separate issue since that doesn't much affect whether or not it can be processed in time.

also a ton of the memory use was (and is) due to inefficient sql queries.

[–] DreadTowel 2 points 1 year ago* (last edited 1 year ago)

I guess that works as an emergency measure. Persistent storage doesn't affect whether the updates are processed in time, but it would act as a sort of swap to keep the memory usage manageable.

For scalability, perhaps, you could run dijkstra and route the updates using the shortest path to each federated node, in a multicast sort of way? That would make the updates scale in a O(log(N)) way, provided that activity isn't too centralised. It would also be great to run periodic "deep scrubs" between instances to sync up each other's activities and provide actual eventual consistency. I guess that's kind of a liberal interpretation of ActivityPub, but I think that's the only way to ensure real scalability.