So I've been troubleshooting the federation issues with some other admins:
(Thanks for the help)
So what we see is that when there are many federation workers running at the same time, they get too slow, causing them to timeout and fail.
I had federation workers set to 200000. I've now lowered that to 8192, and set the activitypub logging to debugging to get queue stats. RUST_LOG="warn,lemmy_server=warn,lemmy_api=warn,lemmy_api_common=warn,lemmy_api_crud=warn,lemmy_apub=warn,lemmy_db_schema=warn,lemmy_db_views=warn,lemmy_db_views_actor=warn,lemmy_db_views_moderator=warn,lemmy_routes=warn,lemmy_utils=warn,lemmy_websocket=warn,activitypub_federation=debug"
Also, I saw that there were many workers retrying to servers that are unreachable. So, I've blocked some of these servers:
commallama.social,mayheminc.win,lemmy.name,lm.runnerd.net,frostbyrne.io,be-lemmy.org,lemmonade.marbledfennec.net,lemmy.sarcasticdeveloper.com,lemmy.kosapps.com,pawb.social,kbin.wageoffsite.com,lemmy.iswhereits.at,lemmy.easfrq.live,lemmy.friheter.com,lmy.rndmm.us,kbin.korgen.xyz
This gave good results, way less active workers, so less timeouts. (I see that above 3000 active workers, timeouts start).
(If you own one of these servers, let me know once it's back up, so I can un-block it)
Now it's after midnight so I'm going to bed. Surely more troubleshooting will follow tomorrow and in the weekend.
Please let me know if you see improvements, or have many issues still.
Thanks for your work and sharing results!
I think that kbin and lemmy are going to ultimately have to record per-instance response time and back off on a given instance. Like, if another instance is failing or overloaded, it's going to have to reduce the frequency with which it attempts to communicate with that instance, to avoid having a ton of workers tied up trying to communicate with that instance.
I'd probably recommend exponential backoff with a low max retries
Ideally, multiple instances could band together and create something like a hub that they all push and pull from. It's a little more centralization, but would likely significantly reduce overall network and CPU consumption.