Welcome to the Python community on the programming.dev Lemmy instance!
Past
November 2023
October 2023
July 2023
August 2023
September 2023
The huge feature of scrapy is it's pipelining system: you scrape a page, pass it to the filtering part, then to the deduplication part, then to the DB and so on
Hugely useful when you're scraping and extraction data, I reckon if you're only extracting raw pages then it's less useful I guess
Oh shit that sounds useful. I just did a project where I implemented a custom stream class to chain together calls to requests and beautifulsoup.
Yep try scrapy. And also it handles for you the concurrency of your pipelines items, configuration for every part,...