this post was submitted on 16 Dec 2023
96 points (96.2% liked)

Programming

17313 readers
9 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]



founded 1 year ago
MODERATORS
 

So, in the era of increasingly good AI powered tools and general search engines full of SEO spam, last week I started creating something little old school and against the trends.

For now It's a have-fun-and-find-out project that main aim is to provide good search results for general web development queries with a special focus on independent blog authors.

The thesis is that no SEO spam website is in the index, which will already filter out most annoying noise on Google/Bing.

Search results are grouped per type: docs, blogs and magazines (e.g. blog platforms or bigger websites).

For now it's far from being done in terms of having a full index, but in most cases it already replaces my go-to search engine when I'm looking up some stuff during work.

I'm looking forward hearing out what y'all think and if you think it makes sense overall I can only encourage you to post some links to blogs or docs that are still missing in the index. I'm more than happy to add it to the crawler.

Responds like: "nei, total shit, who would need that" also accepted but constructive critique more appreciated ;)

EDIT: everyone many thanks for all your voices and comments. I'm super grateful for all of them and happy that we have such place like Lemmy!

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 1 points 11 months ago (1 children)

How is it specifically dev focused? How will the crawler know that the site or page is dev related?

[–] sznowicki 2 points 11 months ago

The crawler takes only the sources that are defined in the crawler repo (it’s open source, check the github org or kukei-spider).

So in this way it’s “curated” in a sense that it would not add anything else to the index.