433
With all the talk about privacy, people seem to be forgetting that censorship is also a major problem for today's internet.
(lemmy.world)
Many "alternative" search engines are better for privacy, but they are still vulnerable to censorship, because they rely on g**gle and m*crosoft's indices for their search results. This isn't a deep-hidden secret either, many of them disclose what search index they use on the "about" page, for example:
- https://duckduckgo.com/duckduckgo-help-pages/results/sources/
- https://support.startpage.com/hc/en-us/articles/5138782571796-Why-isn-t-a-particular-site-appearing-in-the-results
- https://www.ecosia.org/privacy
There are still search engines that (claim to) maintain their own index. Most surprisingly, br*ve:
Very interesting. Too bad it gives really bad results.
Searching for "pokemon" doesn't show in the first page of results
the official website
the wikipedia page
Instead it shows a bunch of random websites that mention the term including this wepage straight out of the year 2000: http://dvdmg.com/pokemon.shtml
As a second search I tried searching for "best fallout for mods" and its first result was some unrelated meta topic (discussing ai generated content) which happened to use the expression "is this the fallout of [action/event]?".
That Pokemon page is awesome though. Man, I miss the old Internet!
if you look at the repo they give thanks to:
"The commoncrawl organization for crawling the web and making the dataset readily available. Even though we have our own crawler now, commoncrawl has been a huge help in the early stages of development."
There is nothing I can find which says how much of the index is CC and how much is their own; if there's a decent amount of CC, this is originally for researchers etc. it's not the best resource in the world for a search index: https://commoncrawl.org/
That being said, as an independent search engine, it's always good to see people take on the massive task of actually building an index, not becoming a proxy.
I searched for Aleister Crowley and learned all about how he’s the secret daddy of Barbara Bush and her satanic siblings. Seems legit. /s