Beehaw didn't really have that much data, they had a well-organized and decorated set of around 10 communities, they had built up a standard of content for over a year... and it just got smashed by the performance problems of even slight growth of data in the post, comments, and federated communities. Tthe site_aggregates UPDATE bug was hammering away too.. with nobody fixing and digging for similar TRIGGER logic problems that June 4's issue 2910 called out.
I regret that I put too much faith in open source ideals, that surely Issue 2910 was explicit enough - and no June 13 hardware upgrade is going to fix such faulty logic. There's two active people who were the ones who approved the code in the first place, they are there every workday, it takes only a few hours - even with testing - to get the faulty SQL trigger entirely removed or fix the logic. I don't know if I'll ever understand how Issue 2910 played out from June 4 to June 30 Reddit deadline, it still seems to be one of the oddest things I've witnessed... to see Beehaw in constant turmoil week after week over what amounts to an easily patched-out or fixed fault in the foundation of the code.
If I could go back to June 4, I would have better focused to this development project had been going on for well over 4 full years of active development and operations of lemmy.ml as an Internet website. I would just keep on eye on that one single Issue 2910 and take on a part of adding some tools to better document the performance of the API.
It crossed my mind, June 4 inspired me to install BulletinTree.com testing main GitHub code merges on June 8, but I painfully see now that should have focused on a small project that came up that most people ignored, but I didn't... right there on June 7.... https://github.com/derivator/tafkars
I regret I didn't take the idea of what tafkars had built and make a proxy agent for the Lemmy API. Start metering the Lemmy API performance response and format the data so that visualization tools could be used. maybe even add nginx proxy log processing as an alternate means to capture data if people didn't want to put an API proxy in place. But there are things the API proxy could do: respond to API calls with a useful error instead of just leave clients with no response or nginx timeout. And the API could even be coded as an emergency-fallback to go pull data directly form PostgreSQL if needed. And building it like tafkars does would require no disturbance of the two developers who had been building lemmy_server for over 4 years.
How did I not see this coming? on June 4 I thought that by June 26 such easily fixed issues like 2910 would be addressed and rolled out... the constant fires burning over at Beehaw were flashing red and the slow-motion train-wreck of it all kept playing out. Beehaw did their best to keep their chins up, but it had to have been a bad experience to welcome people and have it all unfold the way it did.
I canceled my plans for a general purpose Lemmy social media site, the code and network peers were so unstable because of Issue 2910 and similar problems. I had hoped to have a general BBS system, which is what I named Bulletin Tree (BBS reference) for. And instead I saw Beehaw which had been running for 17 months blowing up and having to put up new error messages on their home page...
regrets, no fun :(