Asklemmy

I'm not sure. At a user level perhaps some sort of tracking of logins, posting frequency that sort of stuff. If a user signed up and immediately starts making hundreds of posts, something is probably up and an admin should be made aware somehow. If a dormant account wakes up and starts posting a lot, maybe an admin should take a casual look. Also, as much as people seem to hate it, track some IP addresses, at least temporarily. If 100+ accounts all sign up from one IP in the space of an hour, they are probably less than legitimate.

Assuming the problem is posts and comments by bots there could be something that looks for known spam copypasta, previously moderated/admin'd content, or keywords could be enough on a small instance. Going further perhaps something that reads the posts from users of your instance, has them classified based on previous admin actions (and probably some manual work to flag things as "known good"), and trains some sort of classifier (bayes/markov/ml/whatever). Such tools already exist and are in wide use for email spam filtering and the like. They aren't perfect, but would make an ok first line of defense that can raise things to the attention of the admin.

I am sure you could go further down the automation side, but I would imagine all of these are "human in the loop" sort of things. Once a user/post/whatever gets flagged it generates some sort of report for an admin to take a look at. I don't know how much of this stuff like automoderator or mod bots did on reddit, but a decent amount of it would probably be transferable however it was done.

Perhaps some/all of this doesn't get put into Lemmy itself but can interact through admin APIs and/or the database. I would start at just basic things in Lemmy itself as at the moment there is hardly any admin interface to Lemmy at all. If I just want a list of the users on my instance I have to query the database. Make deleting/purging users easier (I have heard from some admins having bot trouble that it was easier to vs than delete them). Properly split out the modlog per community, show all the details of the action, and show whether something was a mod or admin action.