FYI, bots and crawlers can simply ignore your robots.txt entirely. This is probably common knowledge around these parts, but I've run into clients at work who thought it was a law or something.

I do like the idea of intentionally polluting the data robots will see, as suggested by this comment. There's no reliable way to block them without also blocking humans, so making the crawled data as useless as possible is a good option.

Just be careful not to also confuse screen readers with that tactic, so that accessibility is maintained for humans. It should be easy enough if you keep your aria attributes filled out appropriately, I imagine.

Pollute your site with nonsense that’s invisible to users. Things like pages that are linked to with invisible links that are just walls and walls of random text.

Good luck.

I’m curious about how to verify that these bots respect the rules. I don’t doubt that they do, since it might be a PR nightmare for these big tech companies if they don’t, but I don’t know how to verify them. Asking because I’m also doing this for my website.

By the way, LLMs are usually also trained by common crawl, (not sure to what extent), but I’m not sure whether you want to block common crawl.

Another thing to consider is whether your website is indexed and crawled by web archive, and whether web archive has some policy on AI bot crawlers and scrapers.

Block everyone but the crawlers you like. Blacklists are less reliable than whitelists

Maybe there's some IP address ranges to try block?

It's difficult because, for example, blocking the addresses OpenAI's crawlers use may inadvertently block addresses from Azure used by Bing or whatever.

User-agent: *
Disallow: /

tbh

Pehaps the user (or in this case the bot) will not go directly to your website, but first to some method of captcha verification or something like that, or like those pages (SteamDB for example) that do not open directly but first open a blank page to verify your network and browser with a captcha.