We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.
this post was submitted on 07 Feb 2025
257 points (99.6% liked)
Fediverse
29797 readers
1959 users here now
A community to talk about the Fediverse and all it's related services using ActivityPub (Mastodon, Lemmy, KBin, etc).
If you wanted to get help with moderating your own community then head over to [email protected]!
Rules
- Posts must be on topic.
- Be respectful of others.
- Cite the sources used for graphs and other statistics.
- Follow the general Lemmy.world rules.
Learn more at these websites: Join The Fediverse Wiki, Fediverse.info, Wikipedia Page, The Federation Info (Stats), FediDB (Stats), Sub Rehab (Reddit Migration), Search Lemmy
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Interesting! Is this over a Git issue somewhere? That could explain quite a bit.
might be in relates to issue link here
It was a good read, personally speaking I think it probably would have just been better off to block gotosocial(if that's possible since if seems stuff gets blocked when you check it) until proper robot support was provided I found it weird that they paused the entire system.
Being said, if I understand that issue correctly, I fall under the stand that it is gotosocial that is misbehaving. They are poisoning data sets that are required for any type of federation to occur(node info, v1 and v2 statistics), under the guise that they said program is not respecting the robots file. Instead arguing that it's preventing crawlers, where it's clear that more than just crawlers are being hit.
imo this looks bad, it defo puts a bad taste in my mouth regarding the project. I'm not saying an operator shouldn't have to listen to a robots.txt, but when you implement a system that negatively hits third party, the response shouldn't be the equivalent of sucks to suck that's a you problem, your implementation should either respond zero or null, any other value and you are just being abusive and hostile as a program
Thank you for providing the link. Im actually on GoToSocials side on this particular one. But I wish both sides would have communicated a bit more before this got rolled out.
Yep!