this post was submitted on 01 Jul 2023
941 points (97.7% liked)
Technology
60129 readers
3351 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Text feed are the lightest weight most cachable thing you can serve. The costliest part of the text component is the mixer that ranks the content. The companies scraping them don't care about the ranking they just want bulk tweets. That's what the API is for. Elon charged them insane rates so they all went off the API that cost Twitter a tiny fraction to serve and instead the API consumers switched to crawling the website instead, which costs Twitter orders of magnitude more, but is free for scrapers. Elon is indeed a stable genius.
Holy shit, I hadn't stopped to realize your point there!!! Of course any AI scrapers would just start bit crawler accounts, if any of them weren't already doing that as well. Along with any other info scrapers out there - I can only think of the one example tbh