submitted 2 months ago* (last edited 2 months ago) by lwadmin to c/lemmyworld

What’s the Problem?

Over the past few weeks we have been receiving large amounts of activity requests from kbin instances, primarily kbin.social. The issue is intermittent and the root cause is unknown. It appears as though some process occasionally gets stuck in a loop on the kbin side. For example, a legitimate user may upvote a post in one of our communities, causing kbin to send an upvote action, then an undo action, then an upvote action again, an undo action again, and on and on.

This is not straining lemmy.world’s infrastructure at the moment. However, it does create enough network traffic to be problematic for the Lemmy network in general because these activity requests get passed on to other instances. Some instances have already struggled to keep up with normal traffic from lemmy.world due to how Lemmy processes activity requests (see https://reddthat.com/post/15383278, for more info). When this kbin issue occurs it multiples our outbound traffic and makes the problem worse. The additional activity requests can easily triple the number of outgoing activities from lemmy.world to other instances.

Short-Term Fixes

As a temporary fix we are currently blocking all incoming activity requests from kbin.social.
Users on that site are effectively "read-only" in lemmy.world communities at the moment.

We are hoping to replace the current block with a rate limit on incoming kbin activities. Unfortunately, it may take a few days before the lemmy.world infrastructure team has time to implement that.

We were initially banning the individual users whose actions were being repeated, with the intention of unbanning them when the repeated requests stopped. However, the problem has persisted, and that game of whack-a-mole is manual and too time-consuming. It also gives the impression that those users had done something wrong, which is not the case.

Long-Term Solution

We have reached out to Ernest, who is the kbin dev and kbin.social admin. So far we have not received a response. Ultimately, this appears to be a bug on the kbin side. Hopefully he will be able to dig into this issue soon and resolve it. Until then, we will try to mitigate it on our end as best we can.

We apologize for the inconvenience. Thank you for bearing with us.

For some recent history regarding this issue please see these comments:

top 15 comments
sorted by: hot top controversial new old
[-] [email protected] 47 points 2 months ago

Thanks for the update

[-] testeronious 36 points 2 months ago

maybe add "due to federation issues" at the end of the title.

It might sound like too much, but some people believe that lemmy.world is some sort of dictatorship instance that defederates with every instance they dislike, so these people might read the title and cream their pants thinking their fantasy has come true.

[-] kuneho 3 points 2 months ago

then rather 'technical issues', since there's no problem with the federation side of things, it's just a software bug.

[-] testeronious 22 points 2 months ago

For some recent history regarding this issue please see these comments:



You linked the same comment twice

[-] lwadmin 14 points 2 months ago

fixed, thanks

[-] [email protected] 13 points 2 months ago

Are you getting the same issue from mbin? I'm using kbin.run.

[-] MrKaplan 27 points 2 months ago

so far this has been a single case with kbin.earth and lots and lots of cases with kbin.social.

no other instances have been observed behaving like this yet.

[-] testeronious 8 points 2 months ago

And of course, thank you for the updates :)

[-] [email protected] 4 points 2 months ago

@lwadmin hi, I will try to reach out Ernest to faster fix it.

Thanks for information!

[-] MrKaplan 5 points 2 months ago* (last edited 2 months ago)

feel free to reach out to me directly via matrix at @mrkaplan:lemmy.world if you want

edit: fyi, mentions of @[email protected] will usually not be seen.

[-] [email protected] 0 points 2 months ago
[-] [email protected] 0 points 2 months ago

Interesting. I still can upvote/downvote.

[-] [email protected] 0 points 2 months ago

I'm guessing it's only here on Kbin. For instance I don't see your comment on the non-Kbin side (more>open original url) of this thread (see comment count difference). Then again, federation issues with Kbin is not new either so it is hard to tell (it could just take a day for it to send over). But in this case I'd guess "all actions" does include comments/threads.

Either way I'm not sure there's even a way for you to plainly see that voting is not federated, at least I haven't seen anything like threads from a specific instance that don't have voting arrows or give an error etc.

[-] [email protected] 0 points 2 months ago

Not seeing post is a different issue. It was always the case that the posts propagate with delay.

[-] [email protected] 0 points 2 months ago

You'll see it on the kbin end of things and since I'm here I will too, but no one else will see this from any other instance since lemmy.world won't accept the incoming communications and won't pass it along to any other instances.

this post was submitted on 11 Apr 2024
192 points (99.0% liked)

Lemmy.World Announcements

28601 readers
55 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors


founded 1 year ago