this post was submitted on 05 Jul 2023
59 points (85.5% liked)

Selfhosted

39962 readers
661 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
ruud
Loki
CannaVet
devve
HybridSarcasm
[email protected]
59
lemmony: A better "All" browsing experience for small Lemmy instances (github.com)
submitted 1 year ago by [email protected] to c/selfhosted
18 comments fedilink hide all child comments
 

cross-posted from: https://lemmy.management/post/665809

I made this tool to help self-hosters, new admins, or smaller instances have more global and updated content on their instances.

This is the similar to Lemmy Community Seeder but is designed to be run periodically to capture new communities, and include EVERYTHING by default.

top 18 comments
sorted by: hot top controversial new old
[–] PriorProject 34 points 1 year ago* (last edited 1 year ago) (2 children)

This is a terrible idea, and borderline irresponsible. One of the key reasons that Lemmy doesn't subscribe by default is to avoid forcing servers with many communities to waste time/CPU delivering messages to servers where no one will read those messages. By subscribing to everything, you're telling all those overloaded servers to waste time sending content to your server that you'll never even see.

  • It also will massively inflate your db by multiple GB/day.
  • It will maximize the chances of you downloading and hosting copyright infringing content and content that may be illegal in your jurisdiction but not in the jurisdiction where it's hosted (loli, etc).

It is much MUCH better to just hit lemmyverse.net and subscribe to 10-100 communities you care about. If script accepted a list of community-urls and automated subscribing to those, that would be super nice. Subscribing to the entire lemmyverse is terrible for your server, for your hosting liability, and for the lemmyverse's performance.

[–] [email protected] 11 points 1 year ago (3 children)

The idea of this app, which is desperately needed, is to allow those of us that are running a very small private instance to still get access to all of the communities. If you have a decent user base then you don't need this anymore as the users themselves will provide the functionality.

So in my case I am the only user on my instance so I am certainly not going to be hammering a bunch of instances just to send me updates of whatever total number of communities I'm subscribed to.

[–] PriorProject 17 points 1 year ago (3 children)

So in my case I am the only user on my instance so I am certainly not going to be hammering a bunch of instances just to send me updates of whatever total number of communities I'm subscribed to.

You don't understand how federated replication works. It doesn't occur on-demand when you read a post, it occurs when the instance hosting the community gets a post, comment, or vote. The federation load you place on other servers has nothing to do with how many users are on your instance or how much they read... it has everything to do with how many communities they subscribe to. This script is literally signing you up to proactively receive the firehose of every post and comment in the lemmyverse, without regard for what you actually look at.

I completely understand the idea of the app, and your confusion about how much load it generates is exactly why it's such an irresponsible idea. If you want to fill the timeline of your small instance, do so by subscribing to specific communities you're interested in until your timeline becomes active enough for you. Subscribing to 100 communities you care about will result in a very lively feed of stuff that is interesting to you, while generating a tiny percentage of the federation load this approach does. Carpet bombing the entire lemmyverse with subscriptions you cannot read is madness. It's like writing a reddit app that downloads everything ever posted to reddit to your phone to save you the trouble of picking subreddits to follow. It's bad for reddit, bad for your phone, bad for your isp, and a bad idea all around. If I were running a large instance, I'd defederate with any tiny instance I observed subscribing indiscriminately via this script. It's abuse.

[–] [email protected] 4 points 1 year ago

Subscribing to 100 communities you care about will result in a very lively feed of stuff that is interesting to you, while generating a tiny percentage of the federation load this approach does.

It was a lot of work, but this is what I did (not 100 communities, but enough). I browser all on lemmy.world for days, finding communities I wanted to subscribe to. Then I subscribed to those on my account on my private instance. When I want more communities, I go back to lemmy.world and find more.

[–] [email protected] 3 points 1 year ago (1 children)

I hear ya and I'm open to learning, but your way I have to still use another instance to be exposed to new communities. In essence I can't have a Reddit "all" page... Is that correct?

[–] PriorProject 3 points 1 year ago* (last edited 1 year ago)

There's no such thing as an "all" page in Lemmy or on Reddit.

  • Even on a big server like lemmy.world we don't see everything in the "all" feed. Beehaw defederated with us, and some new/small communities haven't had anyone subscribe to them. These are missing from the "all" feed.
  • /r/all isn't everything on Reddit either. It's missing nsfw subreddits since 2021, all private subreddits, and who knows what else is filtered. It's a curated selection of top posts to form a "frontpage" for those who haven't built up their subscribed feed yet.

On a private Lemmy instance you are the admin and must curate your own frontpage. Community discovery on Lemmy does kind of suck right now, hopefully it will get better over time. For now, Lemmyverse.net is a good place to discover new communities, and you can browse the incomplete "all" feed of a major instance without an account there. Another responder in this thread suggested creating a dummy account to subscribe to stuff you want to see in your all feed but not in your subscribed feed... and it's fine to do that kind of thing liberally. But it should be a human selected list of finite length. Indiscriminate subscription is bad all around.

You're not the first person to want an /r/all, https://lemmy.directory/ tried. You can still see the announcement at https://lemmy.world/post/21875. It is now, less than one month later... broken. I'll leave you to speculate on why and how you plan to avoid whatever problem shut them down.

[–] [email protected] 2 points 1 year ago

You should write something that detects indiscriminate subscribing and automatically defederates with them.

[–] [email protected] 3 points 1 year ago

I also wanted something like this, but I don’t want every single community. I ended up making a not account that I use to subscribe to random communities I might have a fleeting interest in, but don’t want in my subscribed feed.

[–] ericjmorey 0 points 1 year ago

What?

[–] [email protected] -3 points 1 year ago (1 children)

Calm your tits Mr. Key Reasons.

[–] PriorProject 2 points 1 year ago* (last edited 1 year ago)

No, I don't think I will

[–] binkbankbonk 6 points 1 year ago (2 children)

Wait, when I browse ALL from Lemmy.world it’s not showing all posts from all federated instances?

[–] ericjmorey 8 points 1 year ago* (last edited 1 year ago)

To show up in your all listing, a post must be:

  • from an instance that your instance is federated with and has not blocked
  • from a community that someone on your instance is subscribed to
  • from a community you haven't blocked
  • from an account you haven't blocked
[–] [email protected] 2 points 1 year ago

No, technically it is only showing you updates from communities that someone on your instance is subscribe to. In theory if there were a community that no one was subscribed to you and not get any updates nor would you have a show up in all.

[–] [email protected] 6 points 1 year ago

Amazing!!! I need this

[–] [email protected] 5 points 1 year ago

This seems like a bad idea and would only increase load for all federated instances with no real benefit to the community. (Maybe if you were an instance with say 10million users).

However it is exciting and cool, just personally not a recommendation I would be giving.

[–] [email protected] 5 points 1 year ago

Looks great, I'll test this tomorrow

[–] [email protected] 4 points 1 year ago

Thank you very much for the work on this! I just created a topic about this and yours is the most interesting solution!