Everyone now says “block the training bots, allow the search bots”. Most sites doing it have it backwards.

The short version

• 624 sites checked, in four groups, each measured two independent ways.
• 12 were refusing an AI answer crawler while serving a browser normally.
• A later run over 14 agency sites that refuse a crawler at the server found the same thing every time: the rule names only the older user-agents. Not one named a token introduced in 2025 or after.
• So those sites are both stricter and weaker than their owners think. They refuse crawlers nobody meant to refuse, and serve the newer ones that actually fetch a page while an assistant is answering a question.
• Two of the fourteen explicitly allow, in robots.txt, the crawler their own server returns 403 to. Somebody wrote the intention down and something below robots.txt overruled it.
• Separately, 31 sites in 500 did declare a block in robots.txt. Every single one of them was deliberate. None of those is a mistake, and none is a lead.

The split everybody recommends, and why so much of it does not work

The consensus advice in 2026 is a sensible one: disallow the crawlers that gather training data — GPTBot, ClaudeBot, CCBot, Google-Extended — and allow the ones that fetch a page so an assistant can cite it, like OAI-SearchBot, Claude-SearchBot and PerplexityBot. Keep your work out of the training set, stay visible in the answers.

The trouble is that this split is usually implemented once and never revisited, and the crawler roster keeps changing. Claude-SearchBot, Claude-User and Perplexity-User did not exist when most of these rules were written. A list written in 2023 or 2024 names the old bots and silently fails to cover anything since.

That produces the exact inversion of what was intended. In the fourteen agency sites we probed most recently, every single one refused an older name at the server while serving the newer answer-time crawlers normally. The owner believes they are private to AI and visible in search; the reality is closer to the reverse.

And robots.txt will not tell you. Twelve of those fourteen say nothing about AI crawlers at all — the refusal happens at the server, below the level robots.txt controls, usually from a security plugin or a host default. The other two are the sharpest version of the problem: their robots.txt contains a hand-written section explicitly allowing the crawler that their own server then refuses.

We are not naming any of these sites. Every owner has now been written to privately with the evidence and a one-line way to check it themselves. Publishing other people’s misconfigurations before telling them is not what a monitoring company should be doing, and telling them is the entire point.

How we measured it

Two signals, and it matters that they are ranked rather than averaged.

1. What robots.txt declares. A public file the owner controls. Whatever it says is an unspoofable fact, so anything found here is confirmed. It is also a statement of intent, which turns out to be the whole story - see below.
2. A live paired probe. Fetch the same page twice moments apart, once as an ordinary browser and once as the crawler, then compare what comes back. This catches blocks that happen at the server or the CDN, which robots.txt can never reveal because the owner never wrote them.

What the four groups looked like

The finding we did not want

Agencies and online stores are almost all fine. One agency in ninety-five. Zero shops in twenty-five. Publishers block AI crawlers constantly - but they do it on purpose, as policy, and they say so in robots.txt.

We build monitoring for this, so the convenient result would have been widespread breakage. It is not there, and pretending otherwise would be the fastest way to deserve being ignored. What the data actually supports is a duller and more useful claim: almost every site is fine today, and no site has any way of noticing the day it stops being fine. Every clean result above is an argument for watching, not against it.

A robots.txt block is almost never a mistake

Of 500 mid-tail business domains, 31 blocked at least one AI answer crawler in robots.txt. That looked like 31 findings. It was none.

Every one of the 31 also blocked the training crawlers. Not a single site blocked the crawlers that would cite it while allowing the ones that train on it. They were blanket "no AI" policies that a human deliberately typed into a file: about a third news publishers, a cluster of six a near-identical gambling-SEO network, the rest brands and institutions that had made a decision.

There is exactly one robots.txt pattern worth flagging to an owner, and it is the backwards one - blocking OAI-SearchBot or PerplexityBotwhile still allowing GPTBot and ClaudeBot. That site is refusing the crawlers that would cite it while permitting the ones that only train on it, which practically nobody intends. In 500 sites we found zero.

The blocks nobody chose

The paired probe over the same 500 domains produced 42 candidates. Twelve plausible ones were re-tested adversarially - crawler request first, six seconds apart, three different crawler identities - to rule out our own traffic having caused a rate limit. Eleven survived. One was discarded because the browser also started getting refused, which means our address was being blocked, not the crawler.

Then the question that decides what any of it means: does an unknown bot get through? We sent a user-agent we invented - nothing published, nothing anyone could have written a rule about - and plain curl. If those sail through while GPTBot is refused, somebody wrote a rule naming AI crawlers. If everything that is not a browser is refused, the AI crawlers are collateral damage from ordinary bot protection.

Eight AI-specific, three collateral. Sites are not named - see the note on ethics below.

Why we think the owners do not know

On the reasoning above, a rule that names GPTBot is a decision, and telling somebody about their own decision is a good way to look foolish. Except for one thing.

Not one of those eight declares an OpenAI block in robots.txt. An owner who decides to keep AI crawlers out normally says so there - that is what the file is for, and the 31 sites above all did exactly that. Blocking at the edge while staying silent in robots.txt is the signature of a rule the owner did not write.

The clearest case in the batch is an IT reseller whose robots.txt explicitly names and blocks two other AI crawlers and says nothing at all about OpenAI. That is an operator who has thought about this and made choices. Yet GPTBot is refused at the edge, which contradicts their own stated policy. Somebody else put that rule there.

Three failures worth describing

• A host silently rate-limiting one crawler. An agency's site returned 429 Too Many Requests to GPTBot on every single request while ClaudeBot and PerplexityBot got a normal 200 from the same page seconds apart. The response headers identified the hosting platform's edge, not the site's own configuration. Nobody chose that. It arrived with the hosting.
• A site serving AI crawlers 47% less content than it serves a browser - consistently, reproducibly, across three crawler identities. Roughly 231 KB against 440 KB. That can be legitimate. It can also mean half the page is invisible in AI answers. From the outside there is no way to tell which.
• Four sites nobody can settle from outside. Their CDN challenges anything claiming to be a crawler. Their owners cannot answer "can ChatGPT read my site?" by looking, and neither can we, and neither can any other external tool. The only way to know is from inside their own server logs.

The error that cost us a day

The first version of our robots.txt parser reported 15 blocked agencies. All fifteen were wrong. A rule like Disallow: *?s= had everything from the first * stripped off, leaving an empty prefix that then matched every path on the site. An ordinary WordPress robots.txt looked like a total lockout.

It is now a proper RFC 9309 wildcard matcher, validated against four publishers known to block - which it caught - and 95 agencies, which it correctly cleared. We are describing the bug rather than quietly fixing it because the number it produced was confident, plausible, and completely false, and that is the exact failure mode this entire category of tool is prone to. One wrong "your site is blocking ChatGPT" costs more than ten right ones earn.

A note on ethics

No site is named on this page. Most of these owners have not been told yet, and publishing a list of other people's faults before telling them is not behaviour worth trusting with your own monitoring. We are working through them privately. Where a finding is genuinely useful to somebody, they hear it from us first, with the exact command to check it themselves and no pitch attached.

The probe is also deliberately gentle: two requests per site, spaced, with concurrency capped. A company that monitors websites for a living has no business hammering them.

What we would do with this if it were our site

1. Check it once, properly - robots.txt and a live paired probe, not just the file.
2. If something in front of your site refuses AI crawlers and your robots.txt does not say so, ask your host or CDN who put that rule there. In this study that combination was never a decision the owner had made.
3. Then arrange to be told when it changes. Everything in the "fine" column above was fine on the day we looked, which is the only thing a one-off check can ever establish.

Method: 624 sites, checked 27 August 2026. Signal one, robots.txt, parsed to RFC 9309. Signal two, a paired browser/crawler probe of the homepage. Every finding reported here was re-tested with the crawler request first, six seconds apart, across GPTBot, ClaudeBot and PerplexityBot. Sites behind a bot-verifying CDN are reported as unverifiable and are excluded from the block counts. Second run, 29 August 2026: fourteen agency sites that refuse a crawler at the origin, each probed with all eight named crawlers plus three controls — an ordinary browser, plain curl, and an invented crawler user-agent that appears on no blocklist. That third control is what separates a rule aimed at AI from general bot protection: on all fourteen it was served normally. robots.txt was re-read the same morning. We intend to re-run this quarterly; the trend matters more than any single number.