from a quick look it doesn't seem like the crawler uses any federation, it seems to just iterate over the community list api for each tracked instance, it probably doesn't have logic to remove entries that no longer exist, considering that they're still in there.
I couldn't tell you the reason for this, but several posts in [email protected] have been locked by a moderator: https://lemmy.world/modlog/959443
as far as i know, locking a post does not affect voting, only prevents new comments from being federated.
the other example you mentioned, i assume that you're referring to the inconsistency on lemmyverse.net? i haven't looked at how that application works, but it's unlikely to be working with activitypub/federation, instead it's most likely just connecting to various different instances and using their APIs. i've also left a comment over there about that.
i already explained before why these posts don't see votes on startrek.website - there is no local subscriber on that instance. once at least one person from that instance subscribes to the community it'll start seeing updates, which includes votes. there has also been a comment by one of the startrek.website admins about the federation issues caused by them accidentally blocking certain traffic from other instances here.
for discuss.online, there does not seem to have been a longer federation delay according to this dashboard, only about 1.5h delay at some point that was recovered from fairly quickly. it is also very possible that the first subscriber to the community on discuss.online only subscribed after the post was created, as the more recent posts seem to be doing just fine with their vote counts when comparing discuss.online and lemmy.world numbers. looking at our database, i can see the first subscriber to that community from discuss.online joined about 5 hours after the post was posted, which would easily explain the partial votes.
there seem to be two separate issues relating to that.
the number at the top includes "all" communities, including those marked as nsfw.
on a quick glance, it seems all the nsfw marked ones are correctly marked as such, in the sense of also being nsfw on lemmy.
there also are a large number of communities missing overall, but at least the number next to the community tab adds up with the number of listed communities when the filter is set to show nsfw communities as well.
there is also either some kind of data corruption going on or there may have been some strange spam communities on lemmy.world in the past, as it shows a bunch of communities with random numbers in the name and display names like oejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwefoejwfiojwwqpofioqwfiowqiofkwqeifjwefwef which don't currently exist on lemmy.world.
there is indeed a cutoff. there is exponential delay for retrying and at some point lemmy will stop trying until it sees the instance as active again.
there is also a scheduled task running once a week that will delete local activities older than a week. downtimes of a day or two can generally be easily recovered from, depending on latency it can take a lot more time though. if an instance is down for an extended time it shouldn't expect to still get activities from the entire time it was offline.
downtime should not result in missing content when the sending instance is lemmy 0.19.0 or newer. 0.19.0 introduced a persistent federation queue in lemmy, which means it will retry sending the same stuff until the instance is available. depending on the type of down, it can also be possible that there is a misconfiguration (e.g. "wrong" http status code on a maintenance page) that could make the sending instance think it was successfully sent. if the sending instance was unreachable (timeout) or throwing http 5xx errors, everything should be preserved.
we are planning to post an announcement about the current situation with lemmy updates and our future plans in the coming days, stay tuned for that. you can find some info in my comment history already if you are curious.
we're currently aware of delayed federation from lemmy.ml towards lemmy.world and still working identifying the root cause - see https://lemmy.world/post/22196027 (still needs updating that it's happening again).
aussie.zone has been about 6 weeks behind lemmy.world for a few weeks i think at this point, which at least means they're no longer losing activities, but it's still taking ages to reduce the lag.
i don't know what issue there might be with discuss.online right now, but for startrek.website the explanation is rather simple. as you can see in the sidebar, there are 0 local subscribers for the community. when there aren't any subscribers to a community on an instance, the instance will not receive any updates for posts in that community. this includes posts, comments, as well as votes.
startrek.website also had federation issues over the last weeks due to accidentally blocking lemmy instances in some situations.
lemdro.id has recently had some db performance issues that caused it to get around 3d behind lemmy.world, they've been slowly catching up again over the last days.
they did leave before taking the screenshot
Deleting posts or comments is a best effort attempt at removing it from public view.
Due to the federated nature of Lemmy, any time content is posted and sent to other instances, we lose control over what happens next. When content is marked as deleted on our end, this sends a "please delete this" message to other instances, but there is no way to enforce this, as that content is no longer stored on our infrastructure. Once content is publicly accessible on the internet, there is no way to ensure that it gets removed again. Even if all fediverse instances are cooperating and honoring deletion requests, there may be federation issues causing the deletion request being lost, or external actors that just scrape Lemmy or the larger internet to collect as much data as they can, with no chance of the original creator being able to get it purged. Depending on the data, this may no be legal in some jurisdictions, such as EU having the "right to be forgotten" and similar laws.
Additionally, the Lemmy version we are currently using has a bug, where under some circumstances contents of deleted and removed comments are still being displayed to users. While this is expected for mod-removed comments to be visible in the modlog, this is neither expected for deleted comments nor should the comments still be displayed normally. We are looking forward to address this with the next Lemmy update we will be deploying, but we will likely have to build some custom patch for Lemmy to ensure that this does not significantly reduce our moderation experience, as the solution implemented in Lemmy essentially prevents anyone, including site admins and community moderators from seeing removed and deleted comments entirely.
This is further extended by Lemmy having a scheduled task (already in the version we are using here), which runs daily to overwrite deleted posts and comments to remove their contents once they are older than 30 days.
As is right now, if you have sensitive information that you accidentally shared in a comment or post, we recommend editing the contents prior to deletion and removing that.
edit: typo
We will be posting an announcement explaining the current plans and issues in the coming days.
I doubt it
I wouldn't say usually, but they can happen from time to time for a variety of reasons.
It can be caused by overly aggressive WAF (web application firewall) configurations, proxy server misconfigurations, bugs in Lemmy and probably some more.
Proxy server misconfiguration is a common one we've seen other instances have issues with from time to time, especially when it works between Lemmy instances but e.g. Mastodon -> Lemmy not working properly, as the proxy configuration would only be specifically matching Lemmys behavior rather than spec-compliant requests.
Overly aggressive WAF configurations tend to usually being a result of instances being attacked/overloaded either by DDoS or aggressive AI service crawlers.
Usually, when there are no configuration changes on either side, issues like this don't just show up randomly.
In this case, while there was a change on the lemmy.ml side and we don't believe a change on our side fell into the time this started happening (we don't have the exact date for when the underlying issue started happening), while the behavior on the sending side might have changed with the Lemmy update, and other instances might just randomly not be affected. We currently believe that this is likely just exposing an issue on our end that already existed prior to changes on lemmy.ml, except the specific logic was previously not used.
Generally that is true, but when you access a remote community for the first time, Lemmy attempts to backfill several posts from the community. This is limited to only posts, so comments and votes are not included in that. You can also "resolve" a post (or comments for that matter) on an instance from its fedilink (the colorful icon you see next to posts and comments), so when someone links to something elsewhere, a lot of apps will try to open (by resolving) it on the current instance instead, which can also result in posts or comments showing up, even when there isn't a subscriber. Resolving can also be done manually by entering the URL in the search. This seems to not always be that reliable to work on the first try though, so it can help to try again if you have trouble resolving something on the first attempt.
I think there is also something updating community information in the background from time to time, I'm not sure if that only happens under certain conditions or in regular intervals, and I'm not sure whether that fetches new posts at that point either. If it does, it could explain new posts appearing at daily or so interval but without any comments and votes. Backfill should probably only happen initially when discovering the community for the first time though.