this post was submitted on 19 Nov 2024
145 points (98.0% liked)

Lemmy.World Announcements

29075 readers
16 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news ๐Ÿ˜

Outages ๐Ÿ”ฅ

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

Donations ๐Ÿ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 1 year ago
MODERATORS
145
submitted 2 days ago* (last edited 1 day ago) by lwadmin to c/lemmyworld
 

We're aware of ongoing federation issues for activities being sent to us by lemmy.ml.

~~We're currently working on the issue, but we don't have an ETA right now.~~

Cloudflare is reporting 520 - Origin Error when lemmy.ml is trying to send us activities, but the requests don't seem to properly arrive on our proxy server. This is working fine for federation with all other instances so far, but we have seen a few more requests not related to activity sending that seem to occasionally report the same error.

~~Right now we're about 1.25 days behind lemmy.ml.~~

You can still manually resolve posts in lemmy.ml communities or comments by lemmy.ml users in our communities to make them show up here without waiting for federation, but this obviously is not something that will replace regular federation.

We'll update this post when there is any new information available.


Update: Federation is resumed and we're down to less than 5 hours lag, the remainder should be caught up soon.

The root cause is still not identified unfortunately.

all 44 comments
sorted by: hot top controversial new old
[โ€“] MrKaplan 33 points 1 day ago (2 children)

this comment section is not a place to rant about other instances

[โ€“] finitebanjo 3 points 1 day ago

I don't see why not.

[โ€“] Roflmasterbigpimp 1 points 1 day ago

Yeah a swept clean comment section looks way better....

[โ€“] finitebanjo 13 points 1 day ago

I can think of a solution.

[โ€“] greencactus 25 points 2 days ago (3 children)

Ah, I just today in the morning blocked Lemmy.ml. Seems to have been interesting timing :)

[โ€“] [email protected] 26 points 1 day ago (2 children)
[โ€“] Freefall 18 points 1 day ago

In which case, thank you.

[โ€“] credo 14 points 1 day ago

No one said it was a global variable. You can hardly blame the user for poor documentation.

[โ€“] DarkCloud 12 points 1 day ago* (last edited 1 day ago) (1 children)

I've never had a positive interaction with Lemmy.ml. For me it serves as a quarantine space, and a set of pre-tagged users I don't personally enjoy dealing with.

...and I'm not particularly averse to Marxists sentiments either, but they're certainly not good sales people, diplomats, or representative of their cause.

Which is just part of their reputation now. Having a bad experience with a .ml user seems to be part of the lemmy experience. It's kind of comical how consistent it seems.

That said, I'm sure there's good people on .ml.

[โ€“] Pilferjinx 1 points 1 day ago

Yeah, just avoid politics there and it's fine. We all know about their zeal so it's pointless to discuss it.

[โ€“] [email protected] 1 points 1 day ago

Is there a reason people seem to not like .ml? I only joined because it said the instance was for FOSS enthusiasts

[โ€“] uservoid1 18 points 2 days ago (1 children)

Could it be an issue/compatibility with lemmy.ml running Lemmy v0.19.7 ?

[โ€“] MrKaplan 19 points 2 days ago (1 children)

I don't believe it is.

There weren't any network related changes from 0.19.6 to 0.19.7 and we haven't seen this behavior with any of the 0.19.6 instances yet.

The requests are visible with details (domain, path, headers) in Cloudflare, but they're not showing on our proxy server logs at all.

[โ€“] uranibaba 24 points 1 day ago (1 children)

I've read enough posts over at /r/sysadmin, it is always DNS.

[โ€“] scsi 31 points 1 day ago
[โ€“] Deestan 7 points 1 day ago (1 children)

Do these things usually happen from time to time?

I've noticed some lemmy.ml communities looking surprisingly "dead" some days here and there but not thought much of it.

[โ€“] MrKaplan 5 points 1 day ago

I wouldn't say usually, but they can happen from time to time for a variety of reasons.

It can be caused by overly aggressive WAF (web application firewall) configurations, proxy server misconfigurations, bugs in Lemmy and probably some more.

Proxy server misconfiguration is a common one we've seen other instances have issues with from time to time, especially when it works between Lemmy instances but e.g. Mastodon -> Lemmy not working properly, as the proxy configuration would only be specifically matching Lemmys behavior rather than spec-compliant requests.

Overly aggressive WAF configurations tend to usually being a result of instances being attacked/overloaded either by DDoS or aggressive AI service crawlers.

Usually, when there are no configuration changes on either side, issues like this don't just show up randomly.

In this case, while there was a change on the lemmy.ml side and we don't believe a change on our side fell into the time this started happening (we don't have the exact date for when the underlying issue started happening), while the behavior on the sending side might have changed with the Lemmy update, and other instances might just randomly not be affected. We currently believe that this is likely just exposing an issue on our end that already existed prior to changes on lemmy.ml, except the specific logic was previously not used.

[โ€“] [email protected] 5 points 1 day ago

FWIW the communities on walledgarden.xyz have been having federation issues to lemmy.ml for a few days as well.

Since we were/are working through some things with our host I didn't want to bother anyone from lemmy.ml about it, but it's a thing. AFAIK federation is otherwise working normally.

Good luck getting it figured out and resolved!

[โ€“] trespasser69 4 points 1 day ago* (last edited 1 day ago)
[โ€“] AhismaMiasma 2 points 1 day ago (1 children)

Lemm.ee is not loading at all, giving a 503 error. Possibly related?

[โ€“] MrKaplan 1 points 1 day ago
[โ€“] [email protected] 1 points 1 day ago (1 children)

Hmm interesting, so I guess even though I can see this hours old post, my comment should arrive in several days time. Hopefully I haven't responded to anyone on world with anything important recently.

[โ€“] lwadmin 2 points 1 day ago

it arrived a few minutes ago, federation is working again (for now)