I've posted about this before as it relates to mod tools.^1^

The search part isn't all that difficult, there are open source search engines that are easy enough for admins to configure a decent search feature. The more difficult issue is aggregating the data from all our instances to a single source where we can make queries with those existing search engine tools.

I am going to spend some time this weekend working on a proof of concept for a search engine for mod tools. Big picture solution is:

1. Instance admins regularly dump anonymized (i.e. no PII) post and comment data to a public source (possibly torrent, possibly sftp)
2. Other instance admins download each others data and feed it into their search db (e.g. Elasticsearch)
3. Mods & users create tools using this data

BTW: this isn't a novel idea:

• This is what pushshift is for reddit (check out their FAQ/wiki). We're missing mod tools big time and searching/aggregating is huge part of mod tools.
• Up until recently, like last week, Stack Exchange provided a regular dump of their data to the Internet Archive for posterity's sake

EDIT: Linked my OG post on the subject ^1^