Technology

62022 readers

4835 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each other!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed
Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago

MODERATORS

555

Blocking AI bots from Microsoft, others has been “pain in the a**”: Reddit CEO | Huffman says companies must pay to scrape Reddit data even though Reddit itself relies on free, user-generated content (arstechnica.com)

submitted 6 months ago* (last edited 6 months ago) by ForgottenFlux to c/technology

84 comments fedilink hide all child comments

Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren't Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: "Reddit believes in an open Internet, but not the misuse of public content." Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It's unclear how much Reddit's OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

you are viewing a single comment's thread
view the rest of the comments

[+] [email protected] -3 points 6 months ago* (last edited 1 week ago) (1 children)

[removed by mod]

[–] [email protected] 5 points 6 months ago (1 children)

The fact that they managed to restore overwritten posts after people started to delete their history.

[+] [email protected] 0 points 6 months ago* (last edited 1 week ago) (1 children)

[removed by mod]

[–] [email protected] 7 points 6 months ago (2 children)

Reddit used to be open source. There is still a copy of that source available on github. It’s 7 years old so it’s probably significantly different from what they are running now. Still, it gives some insight into the design.

For example, deleted comments aren’t deleted, it just sets a deleted flag. Example code that shows this.

I haven’t dug around the code enough to figure out how editing works, it’s Python code so an unreadable mess. The database design also seems very strange. It’s like they built a database system on top of a database.

[–] ChickenLadyLovesLife 2 points 6 months ago

For example, deleted comments aren’t deleted, it just sets a deleted flag.

FWIW even when you properly delete something from a database table, the deleted row can be reconstructed from the audit tables. And even if that weren't the case, databases are regularly backed up to tape drives or whatever - when people delete or munge all their comments, Reddit doesn't go back into all the backups and make the same changes there. In fact, I would imagine that when they sell their shit to companies for AI training, they sell old pre-AI backups rather than a latest copy.