These issues have been obvious for months, lemmy.ml wasn't sharing the server logs

Now at least there are multiple sites with a modest amount of data who see these issues:

https://github.com/LemmyNet/lemmy/issues/4017

Is there something special about the 28th day of the month and precisely 90 days?

A very obvious server-crashing / denial of service problem was called-out in Lemmy code two days before the Reddit deadline. https://github.com/LemmyNet/lemmy/issues/3394

Observations:

1. Why would anyone think 5 is a good design for production in the first place. It puts into question the developers for over 4 years of experience - they clearly understand the technical issue - it is the same coding / parameter issue for any programming language. What is the motivation / priority here?

2. lemmy.ml developer-run server (then the Lemmy server with the most data) was crashing from PostgreSQL overloads May and June 2023 every day...

3. there were active countdowns to the July 1 Reddit API change, This was June 28.

4. The change takes about 30 seconds to code, by no means is it difficult to understand. But it must be approved by the core developers of over 4 years on the project... and even notify live sites to urgently edit the Rust source code and re-compile. (And why not move this value to an environment variable that can be set without recompiling Rust code?)

June 28 issue opened / code created
July 1 Reddit API deadline
September 28 code published

90 days to change what has contributed to lemmy.ml, beehaw, lemmy.world - and the entire network of Lemmy servers crashing constantly from Lemmy overload. Almost as bad as GitHub Issue 2910 being ignored all month of June 2023!

https://github.com/LemmyNet/lemmy/issues/3394

A change in direction for the project this week?

Maybe the reputation of stability on lemmy.world and people realizing that the amount of activity really wasn't that high - and lemm.ee shutting out images. Most of all, Beehaw's criticism maybe finally resonated.

Beehaw was online a full year before Reddit - and saw just how long-term issues were not being addressed... maybe that is what it took.

It is worth keeping a positive eye on things.

It seems api_tests is unstable, failing around half the time.... been that way for days it seems.

Cambridge Analytica was well underway in 2013, now 10 years ago. People like to think that just because researchers find x on Twitter and y on Facebook - that that is the clearly documented cases - that the tactics and general psychology didn't copy everywhere.

Cambridge Analytica is mostly famous for Facebook... but I don't view their direct targeting of individuals to be the long-term damage. The long-term damage is that they legitimized psycological manipulation, falsehoods, as a form of winning audiences. The were Psychology/Psychiatry professionals who applied human history and experience towards making people believe false things. Like a rebirth of Dr. AA Brill from 1929 on a new scale. The legitimization of it without any ethical uprising...

The only instance with significant creation activity that isn't all bot content... had to resort to cloudflare due to the data performance... and now the problems with that solution have started to be taken on... https://lemmy.world/post/4366376

yesterday .world had to turn off sign-up and even shut down shitposting community.

This is basically the front-door of Lemmy. And as others are starting to notice, 60K users after all the people seeking better and trying out things isn't that many.. And there are seemingly a lot of people who use multiple servers given the technical instability of Lemmy's code... I am one of those people who spends 10 minutes a day on 4 servers, but I'm cutting back because content just isn't there and it's now often gaming topics and news making the rounds as duplicate stories over a 2 to 4 day period (besides memes)

Social media in general... there was so much Facebook hate for the past 8 years... but not much betterment came of it. Today looking over Reddit, it hasn't really changed that much in the past 4 months... there was a group of dissatisfied people who don't seem to want to actually build something better - just want to protest Reddit.

YouTube and TV advertising - that's a huge topic. YouTube does have a lot of small-time original creators, but is the money the reason why? You can''t make money on Reddit or Lemmy unless you have a business shop or something related to specialty topics (such as auto repair in a discussion community about same).

It isn't just the constant database crashes in Lemmy, GitHub issue 2910...

https://github.com/LemmyNet/lemmy/pull/3708

On July 24, sanitation of HTML was added to the code. But the testing was not called for and it broken titles of postings, link parameter ampersands, discussion of programming code in code blocks

It's data... and now it's very difficult to undo all the damaged data that has been put into the database for weeks now.

Lemmy is a Link aggregator, and it damages Links now... the ampersand parameter deliminator in URL links is now broken because of this code not being tested. Why wasn't there a call for testing to something that was going to alter every new post and comment from both federation and Lemmy itself? how did such obvious things such as a ? parameter list in URL get overlooked... and then new bugfix release comes out after this was known as an issue - and still not fixed.

Database crashing that results in lost data from unsaved post and comments, failure to deliver Federation data without any way client or server operators are notified, and damaged data as fundamental as URL website links...

I'm all for code changes gong in fast, but the lack of actually testing things and spot-checking on Lemmy instead of just changing Rust code without really realizing that a link aggregator uses ampersand in URL links... and not asking people to help think of side-effects...

Development process could even ask just a couple sites with more attentive operators to try out the code for a few days and ask people to report any problems before advising all sites to upgrade and break their URL links.

June 4, 2023 is when I felt I had to get involved. And June 8, 2023 is when I crated my own testing-focused experimental Lemmy install that compiled Main from GitHub source and ready to test the changes I thought were surely going to come to the SQL because of GitHub issue 2910. I wanted to test the code that that developers who had been working and running lemmy.ml for over 4 full years - would surely address. It was June 4 with Issue 2910, the June 30 Reddit API cutoff deadline countdown was well under way. Lemmy.ml put in major hardware upgrades on June 13, 2023 - and I was puzzled why such an easy 2 or 3 hour fix for Issue 2910 wasn't put in... but I still had hope that everyone would see the pending countdown to June 30 API deadline and a fix could go in within 10 days - by June 23... for some slack time before June 30..

Watching Beehaw, up and running for 17 months on Lemmy - crashing constantly... I thought, surely the developers were seeing Issue 2910 happening over there... but June 28 came, June 29 came, June 30... nothing. With 4 years of experience on the Rust code base and such, they were the ones to fix Issue 2910.... but from June 4 to June 30, it just didn't happen.

ORM and SQL .. I mean it's lived up to the bad side of the reputation.

It hides it. As someone said here:

"Osiris on June 18, 2021 | unvote | parent | context | favorite | on: DenoDB

I hate ORMs with the fury of a thousand suns. The problem is that I know SQL but now I have to spend a bunch of time trying to figure out how to convert SQL into ORM X just so it can convert it back to inefficient SQL. SQL mostly translates between various databases but ORMs are unique and you have to learn a new API for each one.

I'm on a project using TypeORM and it has been fantastic at helping developers on my team make really bad schemas due to not understanding how to use TypeORM to make the right relationships.

Currently I'm looking at pg-types because you just write SQL and it just helps by making some TypeScript types for you.

(I have used ORMs in C#, PHP, and JavaScript and I hate all of them)."

https://news.ycombinator.com/item?id=27547163

In April and May, Lemmy had a great brand... but server crashes and then lemmy.ml closed sign-up.... sort of confusing people and then Lemmy.world became the brand, then it too has had to close home page many times and constant crashes....

The SQL performance issues that were in lemmy for years really - became trouble once data was introduced.

I don't even know how many times people have come to sing-up and heard about Lemmy to find it slow, crashing, errroring.

Social hazing, rock star glamour attitude among the audience who keeps using Lemmy... but these odd trends in 2023 with social media aren't unique to lemmy, are they?

Threads, Twitter to X, Reddit API change, Lemmy crashing and not really caring about the crashes - and Redis or Memcache to mitigate it.... it's all such an odd year. It's as if everyone stirred the pot with social media but no place to really land upon that isn't crashing or in some form of Elon Musk kind of chaos.

I remember the rise of Reddit, the rise of Gmail - and it just felt like the growth was being dealt with. In 2023, Twitter seems to be leading the entire audience into accepting a kind of crash bad experience and 'stick around, no matter how bad it is' that Reddit users seem to have accepted.

I don't want to be a Lemmy developer... I want the people who know Rust Diesel and such to actually make it work - or choose something else that does work. I want Lemmy to actually not crash all the time and at least be where Reddit was in 2008 when there were just a few programmers doing it.

The SQL statements and server crashes in Lemmy speak for them self, just like the chaos of Reddit and Elon Musk Twitter speaks for itself... but a lot of people accept it. The audience is not asking for stability.

Strange times in 2023!

Unbelievable today the push-back on filtering by what is in-essence, the entire key to lemmy - published date

https://stackoverflow.com/questions/74295698/how-to-make-a-select-where-exists-query-with-diesel

Ok, taking a total fresh approach based on all the months I've been reading people's suggestions and [email protected] kind of things....

All / All Remote / Local / Remote Specific-Isntance

I think server operators should at minimum want the ability to view Remote-only, and even per-instance. Further, I think proxy of API to a community should be something to head towards... where a Lemmy API call can be forwarded as an API call on another Lemmy server for a specific community.

Small, Medium, Large, Trending, Featured Community

Some stock multi-community lists, some of which are dynamically generated like Small / Medium / Large based on number of active posts, users, or other tunable parameters. Encourage people to engage in topics that they normally would not see...

Multi-community sharing

I think foundation is that it should use words and not numbers. Right now, the entire Lemmy system is built upon using localized index numbers for community. Even if it just becomes a JSON blob to throw into PostgreSQL and recall by name...

Maybe have them like communities. And people can subscribe/unsubscribe to a specific list. And the list can have moderators who regulate it (editors). And an option to clone a list to new name.

/mc/ multi-community, name. And no ID numbers. A Trigger or something would have to build the ID numbers in the background.

And a browser of these, much like communities are browsed... and maybe even voting on them. Lemmy doesn't have voting on communities - subscribe alone - but sometimes you don't want to subscribe because they have too much content - but you would still vote for it or recommend it.

Weeks ago I had my moment of facing the attitude of keeping all this secret.

Just casually mention join_collapse_limit was tried behind-the scenes a month ago, then why are there zero post or comments in the entire Lemmy search for join_collapse_limit? I searched the entire GitHub project - no mention of join_collapse_limit. But Ready on the Spot to reveal the secret private communications tried join_collapse_limit log ago.

You know what join_collapse_limit is telling yo8u? Too many JOIN is a performance problem! The entire ORM Rust code and reliance on new JOIN is going to lead to other unpredictable performance problems that varies when there are 10,000 posts vs 2 million posts! And that's exactly the history of 2023... watching the code performance wildly swing based on the size of communities being queried, etc.

What I see is that pull request for ideas get created only after noise is made on a subject. There is a lack of openness to make mistakes in public.

For me,** the server crashes are what annoys me**, not human beings working on solutions. But for most of the people on the project, what seems to anthem is needing to have proper tabs vs. spaces on source code and even adding layers of SQL formatting tools in the middle of what clearly can be described as an SQL performance crisis.

Things keep getting broken: the HTML sanitation took a few hours to add to the project but now weeks of broken titles, HTML code blocks, even URL parameters are now broken on everyday links. The changes to delete behavior have orphaned comments and that has gone on for weeks now.

PostgreSQL data insertion

chaos about big servers crashing from PostgreSQL overload continues...

"Freshness" can also mean frozen fresh ;) Canned fresh. The key concept behind Freshness is to limit the number of posts per community, but also consider that older content if "fresh frozen" is fine for small communities.

This kind of attitude is why I got so upset over GitHub Issue 2910 being ignored for months - and not given specific project call-out/squeaky wheel needs fixed spotlight...

Anyway, PostgreSQL 15.3.1 upgrade to 15.4 now released today, waiting on .deb files.