this post was submitted on 03 Jun 2024
1475 points (98.0% liked)
People Twitter
5190 readers
2063 users here now
People tweeting stuff. We allow tweets from anyone.
RULES:
- Mark NSFW content.
- No doxxing people.
- Must be a tweet or similar
- No bullying or international politcs
- Be excellent to each other.
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Do you understand how hard that is? Most websites do not just let you scrape them entirely because they think you're a competitor. Web scraping involves staying one step ahead of each site. You might as well just pick a site and buy most things there.
Plus you can just have everything delivered. It's easier and you can get access to sales.
Yes, I do indeed, for example the AutoTL;DR bot you might have seen around Lemmy (that does web scraping) is mine.
Sure, you can do an entirely different thing than I was talking about, but then you're replying to something entirely else, aren't you? This was a reaction to a post in the OP which doesn't talk about getting everything delivered.
There are 3rd party APIs that handle scraping.... Eg Red circle API. Problem is they don't really have produce and everything up to date.