Linux

6596 readers

515 users here now

A community for everything relating to the GNU/Linux operating system

Also check out:

Original icon base courtesy of [email protected] and The GIMP

founded 2 years ago

MODERATORS

[email protected]

186

FOSS infrastructure is under attack by AI companies (thelibre.news)

submitted 1 day ago* (last edited 1 day ago) by [email protected] to c/[email protected]

27 comments fedilink hide all child comments

LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 62 points 1 day ago* (last edited 1 day ago) (12 children)

Wow that was a frustrating read. I dd not know it was quite that bad. Just to highlight one quote

they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. [...] If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

[–] [email protected] 25 points 1 day ago (11 children)

the solution here is to require logins. thems the breaks unfortunately. it'll eventually pass as the novelty wears off.

[–] [email protected] 5 points 1 day ago (5 children)

Alternative: require a proof of work calculation.

[–] [email protected] 2 points 1 day ago (1 children)

This is exactly what we need to do. You'd think that a FOSS WAF exists out there somewhere that can do this

[–] [email protected] 2 points 1 day ago (2 children)

There is. That screenshot you see in the article is a picture of a brand new one, Anubis

[–] [email protected] 3 points 1 day ago

Yeah I realised that after posting. I think we need a better one to deal with the cases of letting legitimate users in easier though

[–] [email protected] 1 points 1 day ago

It kind of sucks but it is the best we have for the moment

load more comments (3 replies)

load more comments (8 replies)