this post was submitted on 24 Jun 2023
52 points (100.0% liked)
Lemmy
2172 readers
82 users here now
Everything about Lemmy; bugs, gripes, praises, and advocacy.
For discussion about the lemmy.ml instance, go to [email protected].
founded 4 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
This is in fact my biggest worry of Lemmy's future. People need to be able to search for stuff and I currently don't see how.
I'm doing tests in the next couple days. But I'm trying to build a search engine specifically for Lemmy.
I'm hoping I can open it to the public in a week or so.
Would it be possible to also integrate kbin?
I originally wanted to include the entire fediverse but ActivityPub might have some limitations.
Mainly it appears to be incredibly chatty. So in order for it to work my server, which right now just a RaspberryPi, would need to handle server requests for every single action, or Activity, that any user takes. But long story short every action someone takes on the fediverse creates approximately 1 network request for every instance in the fediverse. With approximately 10,000 instances... you now see the problem.
Now a ray of light in all of that is that I assume Kbin has a public API as well?, and I should be able to create an interface for that. But it's not on my radar for the first release.
Cool! How does it technically work? Does it fetch all titles (and maybe the body and comments) via the api from each instance or do you set up your own private instance and tap into the instance database?
I'm using the public API to grab every post / comment and then I essentially replace the content with only the unique words. Then when you go to search it just looks for any post or comment, in my database, that has the words you typed in. Finally I sort based on the number of upvotes.
Right now it only craws a specific instance that you point it to. But as long as that instance is federated it /should/ get everything. But eventually I plan on using that instance's list of federated instances to scan everything and lighten the load on any one particular instance.
Edit: I thought about tapping into the existing database but the existing database is more geared towards serving content but not necessarily searching. The database that I'm building you can search but I drop so much of the original data that using it for content is worthless.
Now I'm curious what your stack is? Are you using an elastic database?
HTML + JavaScript frontend. Rust backend with a postgres database.
It'll be open sourced once I can get the MVP ready.
Please make sure that you're only indexing Lemmy communities and Kbin magazines (i.e. not microblogs)
In the wider fediverse, there is an actual expectation of privacy beyond "well it's technically possible to scrape everything so we may as well give up". Several people (with reasons of innocent naivete & explicit and blatant malice alike) have tried making fediverse search engines, but all of them are either dead or blocked.
Lemmy/Kbin is in a unique position where global search does make some sense to have, due to it being a public forum focused on topics (and not people), but there is a very real chance that assholes could use an "unbounded" fediverse search engine to find vulnerable people (quite a few of them specifically fleeing to the fediverse to avoid that kind of problem) and harass them.
The concept of privacy within today's Fediverse is asinine and everyone should be pointing that out at every opportunity. Doing otherwise, making believe that some sort of code of conduct or public shame cycle is somehow going to keep people safe, is ridiculous and even more dangerous than a public search engine. By not talking about, very loudly, just how trivial it is to gather this data and how impossible it is to remove it we're sticking our heads in the sand and there will be people who suffer as a result.
Feel free to go do that then
The only ones I found allow you to search for instances and/or communities. I haven't found anything that can search posts, yet. Well DuckDuckGo can but you have to add some elaborate query syntax to make it work. And even then it doesn't know your instance and can potentially pick up duplicates.
sounds promising, can't wait to test it