this post was submitted on 24 Aug 2023
1088 points (97.4% liked)

Lemmy.World Announcements

28641 readers
35 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

founded 1 year ago
MODERATORS
ruud
lwadmin
lwCET
1088
Recent Outages (self.lemmyworld)
submitted 10 months ago* (last edited 10 months ago) by lwadmin to c/lemmyworld
81 comments fedilink hide all child comments
 

Hey everyone, so as I'm sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS'er, and is causing these disconnects to our backend server, problematic for sure.

We've opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

  • LW Infra Team
top 50 comments
sorted by: hot top controversial new old
[–] Cthuwu 267 points 10 months ago (1 children)

As always, the transparency is appreciated. Some growing pains are certainly to be expected

[–] Boozilla 237 points 10 months ago (4 children)

Whenever I get frustrated by the outages I remind myself: still better than reddit.

[–] Shady_Shiroe 89 points 10 months ago (2 children)

Thanks to lemmy I don't doom scroll anymore because either there is an outage or I read all the new content I'm subscribed to in 10-20 minutes

[–] GONADS125 30 points 10 months ago (1 children)

That's mostly how I redditted for years. It was mostly for those moments in between things, on the toilet, laying in bed at night. Not something I did for long periods of time.

[–] chulo_sinhatche 12 points 10 months ago (1 children)

Look at this guy with his reasonable social media use. YOU THINK YOU'RE BETTER THAN ME??

[–] ethman42 7 points 10 months ago

Scrolling a few miles or so a day isn’t too bad, is it?

[–] [email protected] 8 points 10 months ago

Huh, I could keep up with my daily feeds before, now not so much.

[–] malloc 9 points 10 months ago

I fallback to one of the many instances that federate with lw. Don’t get frustrated at all bro

[–] [email protected] 6 points 10 months ago

Yup

[–] RanchOnPancakes 80 points 10 months ago

Maybe its just the times I'm accessing but its seem better this week to me compared to the last few ones.

[–] iforgotmyinstance 43 points 10 months ago

Good luck, no pressure.

[–] perfectra1n 41 points 10 months ago* (last edited 10 months ago)

On your Cloudflare account, if there was a change in the CNAME/A record being proxied vs. DNS only, that could cause an issue, as Cloudflare would then strip headers off the request that your Apache/Nginx would be looking for.

If you enabled HTTP DDoS protection in your Security -> WAF tab (I think that’s where it is) that could do this too. Might be worth disabling.

Also check for any headers your HTTP load balancer might be expecting, that Cloudflare could be stripping.

Might be worth tailing the webserver logs to see what happens to requests coming in from Cloudflare.

[–] Dagnet 37 points 10 months ago

Thanks for keeping us up to date, I don't mind downtime personally as long as I'm aware of what's going on

[–] drewthejedi 36 points 10 months ago (3 children)

Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I've experienced no downtime so far

[–] malloc 17 points 10 months ago

The beauty of a federated app is that I do not have to rely on lw or lemme. Use another instance and go on with my day πŸ˜ƒ

load more comments (2 replies)
[–] mvirts 30 points 10 months ago (1 children)

So what you're saying is we were the ddosers all along 😹

[–] [email protected] 35 points 10 months ago (1 children)

The hack is coming from inside the instance!

load more comments (1 replies)
[–] [email protected] 26 points 10 months ago (3 children)

Let me be real. I never noticed outages stopping. It feels like it's daily, I'm used to it, but I think it happens so often that lemmy.world has lost its growth opportunity, and we alienated the normies. I'm still going to stay on Lemmy, and I believe you're doing the best you can, but we lost for the time being, the migration to Lemmy from Reddit is stunted.

[–] [email protected] 8 points 10 months ago

We shouldn't be trying to grow a single instance. That defeats the whole point of Lemmy. I started on Lemmy world and switched once I got fed up with the constant connection issues. Plus, Lemmy world blocked piracy communities so fuck that. I'm happy that I am able to quickly create an account on another instance.

load more comments (1 replies)
[–] Shadywack 23 points 10 months ago

Only thing that needs to be said here is: thank you for the best Lemmy instance (in my opinion) out there!

[–] webhead 20 points 10 months ago* (last edited 10 months ago)

Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you're getting a hell of a lot more traffic than I am but they haven't blocked Cloudflare on mine yet so that's a plus. :)

[–] TropicalDingdong 18 points 10 months ago (3 children)

Hetzner is a pretty good provider. It would be nice to stick with them if possible.

load more comments (3 replies)
[–] olafurp 16 points 10 months ago

I brows all, never noticed as outage lol.

[–] Cold_Brew_Enema 12 points 10 months ago

You guys rule. Thanks for all the hard work.

[–] Tolstoshev 10 points 10 months ago (2 children)

Is there anything we can do to help? Donations? Tech volunteers? Visit hosting company with a baseball bat?

[–] [email protected] 6 points 10 months ago

I imagine spreading out across instances, and having accounts in instances beyond the biggest one helps, as it reduces traffic and strain on lemmy.world and its servers.

lemmy.world still deserves support, like with donations, though helping reduce the strain is something anyone can do, albeit small, though free of charge.

[–] FlyingSquid 8 points 10 months ago

Thanks for the transparency. I think you lost some trust with the Discord debacle, but this makes me feel a lot better about things.

[–] [email protected] 8 points 10 months ago

The transparency is appreciated.

[–] The_Picard_Maneuver 5 points 10 months ago* (last edited 10 months ago) (1 children)

Thanks for keeping things running as Lemmy grows!

For what it's worth, I'm on all the time and have barely noticed the outages.

load more comments (1 replies)
load more comments
view more: next β€Ί