Hello again all-
First some good news. Our host has finally found the bug that is/was causing us reboots. Things should be much more stable from this point forward and a permanent fix is in the works.
Separately, we're having major issues with federation and we aren't alone. This is where we could really use the help of the fediverse!
Most of the larger instances are several days or even weeks behind our local communities and our host is asking for any information related to this problem. Even instances hosted on the same local hardware are seeing delays from several minutes to several days when these interactions should be nearly instantaneous. If you are seeing these symptoms or anything you feel may be related, please report it here and maybe bring it up with your instance admin. This problem is not isolated to walledgarden.xyz and some instances may not even realize it's happening.
Sincerely, thank you all for your patience and understanding during all the ups and downs. If anyone has anything to add on these topics please don't hesitate to post here or reach out to me via DM.
EDIT 15:25 UTC It appears our communities are flowing out to the larger instances in some capacity again. We are continuing to monitor things and are actively working with our host to get things back to normal ASAP. Thank you for your continued assistance and feedback, it is greatly appreciated.
Update 16:00 UTC The instance was briefly shut down to attempt to correct hundreds of failed federation attempts. We're continuing to monitor performance.
Update 19:35 UTC Another round of adjustments and the instances I'm able to check all seem to be syncing within a minute or less. sh.itjust.works was the most recent to lose sync but is back to normal again. Please report any unusual behavior as you notice it. Thank you!
Update 00:24 UTC Significant headway has been made on the federation issues and we believe we have the solution in sight. ETA is a few hours from the time of this update. Be aware that the federating bug will continue to happen during this time. So far the largest instances affected are lemmy.world and db0. Thank you to everyone who has reached out publicly and privately to share their experiences. It has helped a great deal.
Just as a datapoint, checking in from .world here and while the time of the post isn't localized and is in the future for me I can see this.
Thank you for the reply! Believe it or not even that helps us out.
One thing we are trying to understand is why people from instances like .world are able to post and be seen to us, but our posts/comments/votes aren't reliably making it over there and to many other large instances. Again, this isn't unique to our instance but I suspect not everyone is aware it's happening.
Yo, I think posts federate well with mander.xyz too
From memory the first instances that were being reported out of sync were lemmy.ml, feddit.nl, hexbear.net, and a few others. Today even more like .world, sh.itjust.works, db0 were a few hours to a few days behind.
At the moment it looks like a number of them are working on catching back up. The latest from our host is that all the hardware is back online and performing as expected and things are being monitored. Not everything has magically fixed itself though, so we're still trying to figure it out.