Hello again all-
First some good news. Our host has finally found the bug that is/was causing us reboots. Things should be much more stable from this point forward and a permanent fix is in the works.
Separately, we're having major issues with federation and we aren't alone. This is where we could really use the help of the fediverse!
Most of the larger instances are several days or even weeks behind our local communities and our host is asking for any information related to this problem. Even instances hosted on the same local hardware are seeing delays from several minutes to several days when these interactions should be nearly instantaneous. If you are seeing these symptoms or anything you feel may be related, please report it here and maybe bring it up with your instance admin. This problem is not isolated to walledgarden.xyz and some instances may not even realize it's happening.
Sincerely, thank you all for your patience and understanding during all the ups and downs. If anyone has anything to add on these topics please don't hesitate to post here or reach out to me via DM.
EDIT 15:25 UTC It appears our communities are flowing out to the larger instances in some capacity again. We are continuing to monitor things and are actively working with our host to get things back to normal ASAP. Thank you for your continued assistance and feedback, it is greatly appreciated.
Update 16:00 UTC The instance was briefly shut down to attempt to correct hundreds of failed federation attempts. We're continuing to monitor performance.
Update 19:35 UTC Another round of adjustments and the instances I'm able to check all seem to be syncing within a minute or less. sh.itjust.works was the most recent to lose sync but is back to normal again. Please report any unusual behavior as you notice it. Thank you!
Update 00:24 UTC Significant headway has been made on the federation issues and we believe we have the solution in sight. ETA is a few hours from the time of this update. Be aware that the federating bug will continue to happen during this time. So far the largest instances affected are lemmy.world and db0. Thank you to everyone who has reached out publicly and privately to share their experiences. It has helped a great deal.
