this post was submitted on 23 Mar 2024
16 points (100.0% liked)
Reddthat Announcements
641 readers
4 users here now
Main Announcements related to Reddthat.
- For all support relating to Reddthat, please go to [email protected]
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Right now loading every page -- except profiles -- seems to take about 2~3 seconds with occasional spikes up to ~10 seconds. (Profiles are much slower to load -- ~6 seconds typically and sometimes much longer.) I'm using the "old" (mlmym) interface mainly, but testing with the default interface seems to have similar performance with a lot of spinners on every action. It's not unusable but it's definitely noticeably slow.
Hmmmm that isn't great news. While testing it was working great, which is just my luck.
I can see that there are a lot more spikes now compared to before, and even when ssh'd to the server I can sometimes feel the lag.
I'll be investigating today as I really don't want to roll back to our single server again. There are probably some fine tuning items that need to be done on the network stack side now.
Wishing you the best of luck in your investigation! Hopefully it turns out to be something quick and easy, but having been through the wringer debugging my own projects, I totally get it if it's not.
Alright, I've had a look, and we are now HA. We now have a lemmy front end and back end instances across two different servers with the third server being the new database server.
This has reduced the number of connections through our one server to a more manageable number which seems to have helped with the lag spikes! From the logs it does look like the total number of lag spikes has disappeared but I'm not counting my chickens yet.
With regards to regular requests. With the added server, on average we are looking at
Which again is still not ideal, but it's something to work towards.
One of our servers is also slower at responding in general. So I think we'll decommission it at the end of this month and use the money to buy 1-2 more frontend servers, (that's how big it is).
Thank you for your work
How's the lag today? I still see the long load times in the browser... And it's really getting on my nerves. It's definitely a storage issue now.
I've gone an bought a server in a different hosting company because this is ridiculous. Cue the migration post!
It is noticeable in the browser, but I guess it's definitely an improvement if we are now up-to-date with LW for posts and comments!
Huh. I don't think I got a notice for this reply. So, sorry I didn't respond -- I didn't see it! (Just saw this randomly while checking your profile to see if there was a thread regarding the "next" interface.)
Haven't checked again thoroughly yet today, but the timings I saw yesterday matched your averages pretty well and I definitely saw a reduction in lag spikes after your other reply. I also did a fair bit of poking around with my browser's network inspector -- and I found a potential issue.
It looks like mlmym (the "old" interface -- which is the primary one I use) does not set Cache-Control on pages it serves, so you may be getting traffic to your backend for every logged out lurker and scraper hitting that interface instead of letting your CDN handle it. The header is set for images and such, but not the pages themselves. Lemmy's default UI sets it (
public, max-age=60
for logged out users,private
for logged in users). The "next" interface seems like it sets it toprivate, no-cache, no-store, max-age=0, must-revalidate
for logged out users, which is probably not ideal... I'm not sure yet what the other interfaces are doing since they're JS heavy and I haven't dug into them as much.