1077
AI companies are violating a basic social contract of the web and and ignoring robots.txt
(www.theverge.com)
This is a most excellent place for technology news and articles.
Just thought of a nasty hack the browser makers (or hackers) could use to scrape unlisted sites - by surreptitiously logging user browser history for a crawl list
Perhaps some web extensions already do this and phone home about it.
While there are some extensions that do this, last I saw Google didn't use Chrome for populating Search:
https://blogs.perficient.com/2017/03/15/does-google-use-chrome-to-discover-new-urls-for-crawling/