Community Corkboard

Hello again All! Back with another instance update.

Instance performance and stability

Some of you may have noticed thumbnails that aren't populating correctly, images that don't load, and occasionally even the inability to upload images here. Unfortunately we're dealing with some problems with our Pict-rs service crashing again. This problem is originating at our host and there is nothing we can do about it until the cause is found and corrected. Something upstream is causing a database to corrupt which in turn causes the Pict-rs crash. Our hosting service is aware of it and working diligently to get it corrected.

Other News

Things have been fairly quiet through this holiday season. Here's a quick recap of happenings on WG.

• [email protected] has been seeing more and more users participating and even branched out to more than memes! We were previously maintaining a Matrix room to hang out and BS outside of Lemmy, now we are migrating to SimpleX Chat. You can find the post with a link to join the room here.

• [email protected] has been seeing more action than ever and it's wonderful! Reading people's concert stories and swapping covers of favorite songs has been great. Stop by and let us know what you're listening to!

• [email protected] continues slow and steady growth but we had a great laugh this week when someone new dropped an awesome meme on us. Head on over to see what you're missing!

• [email protected] Just launched this week to celebrate the recently released Pokemon card game on mobile. It's been a real nostalgia hit to see and play with artwork and characters from years ago. The game itself is very addicting too and great fun to play if deck building and pack opening is your thing. You're invited to check it out but be careful, you might just get sucked in!

Finally, a reminder that new member registration at WG is open and we'd love to have a few more people around. The past month or so has been spent getting the site ready for people who might be new to Lemmy and the fediverse but veteran users are of course welcome too. Spend a few minutes checking out the [email protected] section before deciding if this is the place for you.

See you around!

Back with another instance update.

We've just completed the update to Lemmy 0.19.8 which should take care of the pictrs thumbnail problem. As always please contact me if you notice anything out of the ordinary.

This week we also launched a new community- Help and FAQ. I've added a few common things to the community to get us started with hopes that we can fill it out as the instance grows. Our friends around the fediverse are also invited to help populate it with information that might be useful to someone discovering Lemmy for the first time or new to federated social media. Crossposting is always welcome!

[email protected]

Thanks to everyone having fun with us in our local communities! We're continuing slow and steady growth of subscribers and community engagement at [email protected], [email protected], and even [email protected]. If seeing new names is great, seeing return posters and commentary is awesome! You guys are the best and I appreciate you all for contributing in your own ways. 🙏

Finally, a shameless plug that WG is open to new members. I spent some time this week updating sidebars and creating resources for visitors to get a better sense of what kind of place WG is. If it sounds like it might be your type of place we'd love to have you.

See you around!

Another week in the books already! We've had some notable happenings around the instance this week, here's the recap:

Instance Stability and Performance

This has been a big one for us in recent weeks. At the time of posting we are cautiously optimistic that we've finally figured out the root cause of pretty much all of our stability problems and the fixes are already in place.

Federation appears to be updating within seconds across all the instances that had previously reported issues. The solution for this problem involved a deep dive into the architecture with our host and a revised approach to the uptime protection that we use. It's early days but fingers are crossed that we've got it this time.

We're also happy to report that the emergency maintenance at our host has been successfully completed. A temporary fix is in place for the reboot issues that were affecting our Pict-rs and other services, and a permanent fix is being developed. So far so good on this front as well.

Across The Fediverse

We've been fortunate to see growing memberships and participation in a few of our communities even though we've been a rollercoaster of reliability. A HUGE THANK YOU to everyone across the fediverse for the support and engagement! You guys are awesome!

At least one of our communities was featured on [email protected] almost every day last week. At one point the crawler was broken and we made the list by default, this week we made it on merit!

[email protected] has seen some new posters sharing music and comments sharing memories, favorite albums, and more! It's a chill place for sharing and discovering music for everyone and we'd love to see you around!

[email protected] has been slowly but surely gaining traction since the relaunch. Don't be afraid to post your favorite classic memes or your weedbrained OC, we'd love to have more weedtime friends!

[email protected] continues to be our bastion of laziness on the fediverse. Join us in doing the bare minimum to be considered humor on the internet. A place to have fun and be silly just because we can.

Final Notes

A reminder that registration is currently open on walledgarden.xyz for those who are looking for a more relaxed place to call home on the fediverse. Our instance is centered around accepting our differences and learning to coexist without bringing needless negativity to everything. Downvotes are disabled and troublemakers are removed from the instance in an effort to cultivate good vibes across the instance and the fediverse. If that sounds like something you'd like to try our sign up page is just a click away.

See you around the fediverse!

Hello again all-

First some good news. Our host has finally found the bug that is/was causing us reboots. Things should be much more stable from this point forward and a permanent fix is in the works.

Separately, we're having major issues with federation and we aren't alone. This is where we could really use the help of the fediverse!

Most of the larger instances are several days or even weeks behind our local communities and our host is asking for any information related to this problem. Even instances hosted on the same local hardware are seeing delays from several minutes to several days when these interactions should be nearly instantaneous. If you are seeing these symptoms or anything you feel may be related, please report it here and maybe bring it up with your instance admin. This problem is not isolated to walledgarden.xyz and some instances may not even realize it's happening.

Sincerely, thank you all for your patience and understanding during all the ups and downs. If anyone has anything to add on these topics please don't hesitate to post here or reach out to me via DM.

EDIT 15:25 UTC It appears our communities are flowing out to the larger instances in some capacity again. We are continuing to monitor things and are actively working with our host to get things back to normal ASAP. Thank you for your continued assistance and feedback, it is greatly appreciated.

Update 16:00 UTC The instance was briefly shut down to attempt to correct hundreds of failed federation attempts. We're continuing to monitor performance.

Update 19:35 UTC Another round of adjustments and the instances I'm able to check all seem to be syncing within a minute or less. sh.itjust.works was the most recent to lose sync but is back to normal again. Please report any unusual behavior as you notice it. Thank you!

Update 00:24 UTC Significant headway has been made on the federation issues and we believe we have the solution in sight. ETA is a few hours from the time of this update. Be aware that the federating bug will continue to happen during this time. So far the largest instances affected are lemmy.world and db0. Thank you to everyone who has reached out publicly and privately to share their experiences. It has helped a great deal.

Hello again all!

As our communities continue to grow and spread through the fediverse I thought it might be a good idea to get in the habit of providing regular updates about the state of our instance and the communities within. I'll try to remember to post updates every Sunday but if I forget or nothing of interest happens we may skip weeks here and there. Enough rambling, ON TO THE NEWS!

Uptime

The rollercoaster of service uptime was a little smoother this week but we are continuing to work through issues with our host. To begin with we were chasing irregular hardware demands that were causing us huge slowdowns and timeouts, now our host is neck deep in chasing a bug on their side. Things are set to come back automatically after crashing and the reboot usually only takes a few minutes, but the annoyance is very real and very understandable. If you have trouble accessing communities or images on our instance please give it a couple minutes and retry. We are hoping to have the answer on Monday.

Communities

This week a couple of our communities were fortunate to make it to [email protected] for some extra exposure. The bot has been on an acid trip for a week or so and doesn't remember who it is anymore, but we'll take it.

[email protected] is nearing 100 people and seeing a ton more engagement compared just a few short weeks ago. It's great to see people getting in the spirit of the community! I know there's more of you out there with weed humor to share, don't be shy!

[email protected] hit 100 members and kept on rolling! We've got more and more people posting the music they love every day and enjoying the music others have shared. Thanks for keeping things interesting and so varied! Because of you the garden is groovier than ever!

[email protected] is doing the bare minimum to be considered a community, as is tradition. In its current state it's mostly a shitposting echo chamber for a couple of us but others are always welcome to join in!

Misc.

Finally, a reminder that registration is back open on WG if you're looking for a chill place to call home on the fediverse. There is a short application and registration requires approval to prevent bots and scammers from sneaking in undetected but applications are usually approved in less than 24 hours.

Thanks for stopping by everybody!

Hello again all-

Once again we've run into a ceiling that required some tweaking to get the instance running again. I'm very sorry for the continued and unannounced downtime. I know it's incredibly frustrating and your continuing patience is greatly appreciated.

As a byproduct of this craziness our upgrade to Lemmy 0.19.7 has been bumped up from the weekend to now, and it's already done!

As always, please contact me here or via DM with any comments, concerns, or downtime reports.

Thanks!

Hi all,

We're continuing to work through some reliability and stability issues with our host.

The TLDR is that random hardware restarts are causing some services to crash and become unavailable for short periods of time while they're restarting. This isn't something we can do anything about and our host is working hard to track down the problem and solve it.

Most recently and commonly this is affecting our pictrs service which causes images hosted on walledgarden and image previews to become unavailable for short lengths of time. Usually the service is back up within a few minutes.

This is separate from the crashing and slowdowns we were going through last week that appear to have been ironed out. Last week's instability was a combination of organic growth of our communities and higher than normal load on the server for a personal project on the instance. If you're still experiencing extreme or frequent slowdowns on walledgarden communities, please report it to me via DM so I can get it resolved.

The update to Lemmy 0.19.7 is scheduled for next weekend. Please note this will take us offline for a short time, hopefully not longer than 30 minutes.

As always, feel free to contact me if you notice anything out of the ordinary and I'll see what I can do to sort it out. Thanks all!

Sorry for the announced reboots.

We're still working through some things trying to get everything tuned correctly. If things go dark just retry in a minute or two and we should be back.

I'd also be interested in hearing if anyone is experiencing errors or delays when browsing or posting that seem out of line with the rest of Lemmy. Sorry it's such a pain!

Please don't let this discourage you from participating in the communities. If anything now is the best time to tax the server. Thanks for all your patience.

Update on instance performance upgrades

The server is running on double the RAM it was previously which should help with the slowdowns. [email protected] in particular is really putting the hardware to work from all the growth which is great to see!

Federation Desynchronization

A few instances have fallen out of sync with walledgarden.xyz so some posts, votes, replies, etc may be lagging behind. This should sort itself out as federation syncs back up assuming we aren't rate limited somewhere.

If you notice anything unusual please don't hesitate to post about it here or DM me.

Thanks for hanging out with us!

Greetings all.

The last few weeks have seen a lot of new things for our community, and with it have come some hiccups and growing pains. While I think most of the major issues are behind us we seem to hit snags here and there that I could use the fediverse's help to get ironed out.

If you happen to notice slow downs, timeouts, errors, or other anomalous behavior related to walledgarden.xyz please report it here or to me via DM.

I've been working with our host to ensure the smoothest experience possible but as we continue to grow and try new things we've run into some issues and sorting them out isn't always immediate. Things like losing federation sync to random instances are hard to keep track of without help from people using our communities reporting it. If you are a regular user of [email protected], [email protected], [email protected], etc please speak up when you run into problems so we can get them sorted.

THANK YOU ALL for your contributions and support in our communities, we'll see you around the fediverse!

We will be waiting approximately 2 weeks (probably slightly less) before updating to the latest Lemmy version. I'll try to post a warning before things go down but it isn't expected to take long.

Apologies for the recent downtime, our instance was being migrated to upgraded storage. Thanks for your patience and understanding!

Since this instance is very new there isn't a lot of data cached on our server. As more of you join and add your own subscriptions to the community, we are being rate limited by other instances. This is a natural part of how federating works currently and it will sort itself out. Growing pains.

Depending on the communities being joined it can take anywhere from a few minutes to hours or even a day or so to smooth back out. It isn't anything to worry about, join whatever you feel like and the servers can sort it out. There isn't anything we can do about it.