Announcements about system updates or other things related to this instance.
Seems federation is still not fixed even though it certainly helped to federate some things, while leaving others... :/
On the positive side, the pull request that may fix federation properly in the software was approved by one of the Lemmy devs 2 hours ago so it should be merged with the main code soon:
I am reading this from lemmy.fan so you're definitely federating now!
Thats great. :)
I know other instances on 0.19.1 had issues with federation so I can share what we did here to try and fix it.
There is a database table in postgres called instance and in our case, it had thousands of instances that were considered dead. Our instance didnt seem to try and talk to them anymore since the "updated" column would never update.
So we ran a simple database query to set the updated column to now() for all instances that we knew about:
update instance set updated = now()
This caused Lemmy to go out and try to talk to all of the instances again, and reconsider if they were dead or not.
According to this site we are now federated with 3000+ instances again:
https://phiresky.github.io/lemmy-federation-state/site?domain=lemmy.today
I hope we stay federated this time. Will keep an eye on it. :)
Yay, thank you admin! Will test it out
Hmm, I made some recent posts to remote communities and so far they havenβt shown up on any other instances yet, but most of my recent comments at least seem to be visible.
Thanks for your hard work trying to figure this out and happy 2024.
If you feel comfortable sharing a post that hasn't federated, please link to it. :)
Itβs not like theyβre private anyways, but sure:
https://lemmy.today/post/4778593 https://lemmy.today/post/4786719
Neither of these seems to be visible on Lemmy.world
Yes, it seems federation still is buggy. :/
Your comments are on lemmy.world now but they needed a Lemmy software restart to make them federate:
https://lemmy.today/post/4778593 => https://lemmy.world/post/10236722
https://lemmy.today/post/4786719 => https://lemmy.world/post/10236757
So still gremlins in the software...
They should be there now - it seems a restart of Lemmy is still required for posts to federate....
For me Lemmy was working very well yesterday though, and I got responses to posts and comments right away. However, it seems like federation stops to work after a while again. The site I linked before:
https://phiresky.github.io/lemmy-federation-state/site?domain=lemmy.today
That one was showing that we had thousands of lagging instances and no updated ones just a few moments ago. And after a restart, they are all up to date again. So absolutely some bug remaining...
This fixed my federation issues as well. Thanks a bunch! I am however missing a lot of comments from this thread (Like the one from db0).
Yeah its completely empty on https://lemmy.dbzer0.com, even after a restart of the server here.... :/
Federation is really being weird since 0.19... But at least you managed to offer a bit of hope!
Yeah I think it will be fixed soon. Just need to wait a bit longer for Lemmy devs to push some code. :)
Turns out this only worked for a bit. I had to redo the fix today.
Yep I had to redo the fix also once more and also split up Lemmy in several docker containers. One container is doing the federation part only and I can restart that without disturbing the Lemmy user interface or database. So that one is restarting every 10 minutes now but we don't notice it since it's happening in the background.
Anyway, I read in the github thread that a fix for federation is working and will come soon... Just need to wait a bit longer.
Thanks for your insight! I've updated to 0.19.2 and stuff seems to be on the up and up federation wise. Is the fix merged already ?
Yep here is the release notes also:
https://join-lemmy.org/news/2024-01-10_-_Lemmy_Release_v0.19.2_-_More_Federation_Fixes
Cheers!
Is this the 19.2 or a 19.2-bis?
If so is there a docker image of it :-) ?
Interestingly, I see your OP, but I don't see your comment. However your instance's updated column is current in my instances table
Yeah that post looks very empty on https://lemmy.dbzer0.com, and even after I restarted Lemmy here, none of it federated to the https://lemmy.dbzer0.com instance. But I can see that other posts and comments has federated to other instances after a restart, like these ones to Lemmy.world:
https://lemmy.today/post/4778593 => https://lemmy.world/post/10236722
https://lemmy.today/post/4786719 => https://lemmy.world/post/10236757
I get the feeling that there are many small bugs in the software....and getting a clear picture of why some things federate and some doesnt is difficult of course.
@mrmanager Tried following the community from Mastodon but it says pending. I'm not sure this entirely fixed it.
Maybe it worked now after a restart of the server?
@mrmanager Doesn't seem to be fixed by a restart unfortunately.
I guess we just wait for the software fix. Something must be wrong there...
@mrmanager Yup something is definately wrong here. Made a Github issue so hopefully it can be addressed soon.
I've been seeing federation issues again. For example:
https://lemmy.today/comment/4640480
That hasn't made it to sh.itjust.works, and I made the comment 4 hours ago.
Not sure if this is the same problem or something new.
Yeah we need the new 0.19.2 version of Lemmy to hopefully fix things. Good news is that I saw it being worked on yesterday on github, so it should come very soon now. :)
Yep finally it works, getting responses to comments again!