I made a Lemmy instance with a custom algorithm that keeps only the top 20% most unique (=interesting?) posts. It does this by calculating a similarity score between every post on my instance and all posts that came before it. The top 80% of posts with the highest self-similarity get removed instantly.

The idea would be that this allows me to cut through the noise that's running through the communities, similar to how xkcd-signal attempted to do 20 years ago.

The instance is mostly meant for reading, not posting. So it has a very open federation policy (for now).

If anything, this is experimental. So please let me know what you think! You can see the type of stuff that gets removed in the modlog (https://lemmy.coffee/modlog).

top 31 comments

sorted by: hot top controversial new old

[–] AbouBenAdhem 28 points 3 days ago (1 children)

If there are a bunch of posts on a particular topic, shouldn’t it keep at least one of them? Otherwise it would tend to completely filter out the most significant or interesting topics.

[–] [email protected] 2 points 12 hours ago

Yeah, that's one of the potential issues that I'm currently looking out for. So far the main thing I can tell is that memes get removed like crazy (https://lemmy.coffee/c/[email protected]?dataType=Post&sort=New) and the posts on the homepage are generally much less meme-intensive when compared to instances like lemmy.world or lemmy.ml.

[–] [email protected] 19 points 3 days ago* (last edited 3 days ago) (1 children)

Interesting. One of my instance's guiding philosophies is "Quality over Quantity". I've taken different steps toward achieving that (defederate from the Reddit repost instances, disallow pretty much all content bots, manually/locally mod duplicate posts, etc).

Do you plan to publish your algorithm/filter? Would be interested in seeing if it could be tuned and possibly reduce some of the workload for me.

[–] [email protected] 8 points 3 days ago (1 children)

Do you plan to publish your algorithm/filter?

In an ideal world sure. But I'd have to think about that some more, because in principle I don't want people to game it :)

[–] [email protected] 11 points 3 days ago (2 children)

Lemmy's license is AGPL, so you would need to at least publish changes to Lemmy itself 😉

(I don't know if e.g. the code for the algorithm is separate, in order to have a closed source algorithm with an open source Lemmy fork)

[–] [email protected] 1 points 12 hours ago

I made no changes to the lemmy codebase, its all done through an auto-moderating bot that auto-removes posts that don't meet the standard :)

[–] [email protected] 2 points 3 days ago (1 children)

Does GPL/AGPL require you to publish the code even if you are not selling the software? As in I could run a library computer with my custom Linux distro without giving anyone the source, but I wouldn't be able to publish it or sell it only as binary blobs, right?

[–] [email protected] 9 points 3 days ago (1 children)

Selling is outside the scope of the licence, you can do whatever you want with monetisation, be it free or paid-for.
But any one person that uses your GPL if local, AGPL if local or through a remote service, has the right to request you a copy of the code and you have an obligation to comply and provide it

[–] [email protected] 2 points 3 days ago

Thanks!

[–] [email protected] 9 points 3 days ago (1 children)

Oh, kinda like r9k!

[–] [email protected] 6 points 3 days ago

That's the idea, yes.

[–] anon6789 7 points 3 days ago (1 children)

I was curious what would happen to the ratio of political posts, specifically Trump/Elon, to other communities, but it feels >= the amount as All on lemmy.world.

None of my superb owls look to have made it through, but I didn't see them removed in the mod log. We're a pretty large community, so I'd have thought some would have gotten through. I don't recall if I saw stuff from any of the animal comms.

[–] [email protected] 2 points 12 hours ago (1 children)

I'll take a look at [email protected]!

The political posts thing is interesting. So far, mainly news posts with text descriptions remain on the front page (which I personally prefer over the memes, but that's just my preference). I am having a hard time deciding if I actually find them interesting though.. ^^ I think I actually prefer it over my reddit homepage, there tends to be much of the same, usually.

For comparison, see: https://imgur.com/a/xbzMXmQ

[–] anon6789 1 points 10 hours ago (1 children)

I do look to have made it to your feed now! It deleted my 3 posts that were photos with short blurbs and kept my news article that is mostly text.

We rarely get the meme type posts, but if your setup is looking at anything like text:photo ratio, that could be doing it.

I have most meme communities block myself, so I understand how overtaking they can be to a feed. 😁

I'm interested in what you're doing because I am interested in news and politics, but it would be nice to get rid of most of the garbage US news and the dupe posts, and let a better variety of news come in. I've gone to keyword blocking, but while I have "Trump" blocked, I'd still like to see things from other countries about their responses to his actions., for example.

[–] [email protected] 2 points 10 hours ago

I do very few things explicitly, I just punish self-similarity in a very specific way. I guess posts with actual text in the body are just more unique, given all previous posts on the instance.

Maybe using the filtered posts as a base in combination with some client side keyword blocking will be useful? The keyword blocking would be much more individual for each user.

[–] [email protected] 3 points 3 days ago (1 children)

I've been through a few pages, could only find this post about Lemmy apps from [email protected] ; https://lemmy.coffee/post/6860?scrollToComments=true , with a single comment (mine).

No posts from [email protected], while it is much more active. Do you know why?

On the other hand, [email protected] , [email protected] and [email protected] seem to be doing fine

[–] [email protected] 4 points 3 days ago (2 children)

I added the larger communities before starting to remove posts, so there may be historical posts still hanging around. Maybe everything from BuyFromEU was deleted?

You can see the kind of stuff that stays best via homepage ALL > Top last N hours

[–] [email protected] 2 points 3 days ago (1 children)

Seems like [email protected] votes and comments aren't federated to your instance: https://lemmy.coffee/c/[email protected]

[–] [email protected] 1 points 12 hours ago (1 children)

They are now :)

If the posts in https://lemmy.coffee/c/[email protected] are more interesting than in https://feddit.uk/c/buyeuropean, I am having a hard time deciding that lol

[–] [email protected] 1 points 10 hours ago

Thanks!

[–] [email protected] 2 points 3 days ago

Like this, for example: https://lemmy.coffee/?dataType=Post&listingType=All&sort=TopSixHour

[–] [email protected] 1 points 3 days ago (1 children)

Cool Idea!, Created an account to check it out.

[–] [email protected] 1 points 12 hours ago

Cool. Let me know what you think!

[–] [email protected] 0 points 3 days ago (2 children)

This is very interesting. I've been thinking of how a similar but different system could be implemented. The front page of any instance always seems to have two to four posts by the same person, and I've been following a rule that if I notice it I block them to remove their clutter of posts from my feed. Unfortunately, most of these accounts are brand new ones posting memes, so it feels like for every two I swat down, four more take their place.

I was looking for an instance that allowed me to mute all new accounts, or less likely an instance that filters out posts by the same person, but it doesn't seem such a thing has been created.

This isn't the same that I wanted, and if I'm not mistaken it only applies to posts inside this instance, since the algorithm is removing them? Or does it filter out all similar posts across the fediverse feed? Still, it's close in concept.

[–] [email protected] 2 points 12 hours ago

From what I can tell, the larger instances have frontpages without much user duplication. They have a LOT more memes than I do tho :)

https://imgur.com/a/xbzMXmQ

[–] [email protected] 2 points 3 days ago (1 children)

PieFed places an icon next to the username to help highlight such aspects of a person's account. The one for "new account, less than 2 weeks old" is very useful. Others need tweaking such as "potential unregistered bot that posts far more often than comments", and "contentious user with far more downvotes than upvotes". I find it useful not to block people but to simply scroll past or to tailor my response to knowing that info.

In addition to PieFed, there are some Lemmy apps that will do this too, although I am not sure which ones (perhaps check out Sync and Connect) - the trick here is ofc to auto apply it to all accounts as you read through your feed.

[–] [email protected] 2 points 12 hours ago (1 children)

I would still argue that thunder is the best app for lemmy

https://github.com/thunder-app/thunder

[–] [email protected] 1 points 11 hours ago (1 children)

Really, if you don't mind my asking, why is that? Or do you know somewhere that offers a feature comparison? I seem to hear people saying that more for Voyager than Thunder, and both are FOSS. My own only super brief look seemed to confirm that in some few ways like display of full images in the former whereas in the latter it will only show like a third and then you have to click and wait for it to load to see the rest. However, I have only tried Thunder without logging in, so I guess I'll give it my account credentials and try it out again:-).

[–] [email protected] 1 points 11 hours ago (1 children)

Maybe I misremember but didn't voyager have ads? That was a turn off for me.

Or maybe that was boost?

[–] [email protected] 1 points 11 hours ago (1 children)

That sounds like Boost. Voyager is FOSS.

I want to like Thunder, and I'm sure there's all sorts of reasons to, I just haven't seen any of them yet:-).

[–] [email protected] 1 points 10 hours ago

My mistake then. I have to give Voyager another try, I guess :)