We checked 624 websites to see how often AI crawlers get shut out

The short version

• 624 sites checked, in four groups, each measured two independent ways.
• 12 were refusing an AI answer crawler while serving a browser normally.
• Not one of the eight clearest cases mentioned OpenAI in its robots.txt - the file that exists precisely to state this.
• Separately, 31 sites in 500 did declare a block in robots.txt. Every single one of them was deliberate. None of those is a mistake, and none is a lead.

How we measured it

Two signals, and it matters that they are ranked rather than averaged.

1. What robots.txt declares. A public file the owner controls. Whatever it says is an unspoofable fact, so anything found here is confirmed. It is also a statement of intent, which turns out to be the whole story - see below.
2. A live paired probe. Fetch the same page twice moments apart, once as an ordinary browser and once as the crawler, then compare what comes back. This catches blocks that happen at the server or the CDN, which robots.txt can never reveal because the owner never wrote them.

What the four groups looked like

The finding we did not want

Agencies and online stores are almost all fine. One agency in ninety-five. Zero shops in twenty-five. Publishers block AI crawlers constantly - but they do it on purpose, as policy, and they say so in robots.txt.

We build monitoring for this, so the convenient result would have been widespread breakage. It is not there, and pretending otherwise would be the fastest way to deserve being ignored. What the data actually supports is a duller and more useful claim: almost every site is fine today, and no site has any way of noticing the day it stops being fine. Every clean result above is an argument for watching, not against it.

A robots.txt block is almost never a mistake

Of 500 mid-tail business domains, 31 blocked at least one AI answer crawler in robots.txt. That looked like 31 findings. It was none.

Every one of the 31 also blocked the training crawlers. Not a single site blocked the crawlers that would cite it while allowing the ones that train on it. They were blanket "no AI" policies that a human deliberately typed into a file: about a third news publishers, a cluster of six a near-identical gambling-SEO network, the rest brands and institutions that had made a decision.

There is exactly one robots.txt pattern worth flagging to an owner, and it is the backwards one - blocking OAI-SearchBot or PerplexityBotwhile still allowing GPTBot and ClaudeBot. That site is refusing the crawlers that would cite it while permitting the ones that only train on it, which practically nobody intends. In 500 sites we found zero.

The blocks nobody chose

The paired probe over the same 500 domains produced 42 candidates. Twelve plausible ones were re-tested adversarially - crawler request first, six seconds apart, three different crawler identities - to rule out our own traffic having caused a rate limit. Eleven survived. One was discarded because the browser also started getting refused, which means our address was being blocked, not the crawler.

Then the question that decides what any of it means: does an unknown bot get through? We sent a user-agent we invented - nothing published, nothing anyone could have written a rule about - and plain curl. If those sail through while GPTBot is refused, somebody wrote a rule naming AI crawlers. If everything that is not a browser is refused, the AI crawlers are collateral damage from ordinary bot protection.

Eight AI-specific, three collateral. Sites are not named - see the note on ethics below.

Why we think the owners do not know

On the reasoning above, a rule that names GPTBot is a decision, and telling somebody about their own decision is a good way to look foolish. Except for one thing.

Not one of those eight declares an OpenAI block in robots.txt. An owner who decides to keep AI crawlers out normally says so there - that is what the file is for, and the 31 sites above all did exactly that. Blocking at the edge while staying silent in robots.txt is the signature of a rule the owner did not write.

The clearest case in the batch is an IT reseller whose robots.txt explicitly names and blocks two other AI crawlers and says nothing at all about OpenAI. That is an operator who has thought about this and made choices. Yet GPTBot is refused at the edge, which contradicts their own stated policy. Somebody else put that rule there.

Three failures worth describing

• A host silently rate-limiting one crawler. An agency's site returned 429 Too Many Requests to GPTBot on every single request while ClaudeBot and PerplexityBot got a normal 200 from the same page seconds apart. The response headers identified the hosting platform's edge, not the site's own configuration. Nobody chose that. It arrived with the hosting.
• A site serving AI crawlers 47% less content than it serves a browser - consistently, reproducibly, across three crawler identities. Roughly 231 KB against 440 KB. That can be legitimate. It can also mean half the page is invisible in AI answers. From the outside there is no way to tell which.
• Four sites nobody can settle from outside. Their CDN challenges anything claiming to be a crawler. Their owners cannot answer "can ChatGPT read my site?" by looking, and neither can we, and neither can any other external tool. The only way to know is from inside their own server logs.

The error that cost us a day

The first version of our robots.txt parser reported 15 blocked agencies. All fifteen were wrong. A rule like Disallow: *?s= had everything from the first * stripped off, leaving an empty prefix that then matched every path on the site. An ordinary WordPress robots.txt looked like a total lockout.

It is now a proper RFC 9309 wildcard matcher, validated against four publishers known to block - which it caught - and 95 agencies, which it correctly cleared. We are describing the bug rather than quietly fixing it because the number it produced was confident, plausible, and completely false, and that is the exact failure mode this entire category of tool is prone to. One wrong "your site is blocking ChatGPT" costs more than ten right ones earn.

A note on ethics

No site is named on this page. Most of these owners have not been told yet, and publishing a list of other people's faults before telling them is not behaviour worth trusting with your own monitoring. We are working through them privately. Where a finding is genuinely useful to somebody, they hear it from us first, with the exact command to check it themselves and no pitch attached.

The probe is also deliberately gentle: two requests per site, spaced, with concurrency capped. A company that monitors websites for a living has no business hammering them.

What we would do with this if it were our site

1. Check it once, properly - robots.txt and a live paired probe, not just the file.
2. If something in front of your site refuses AI crawlers and your robots.txt does not say so, ask your host or CDN who put that rule there. In this study that combination was never a decision the owner had made.
3. Then arrange to be told when it changes. Everything in the "fine" column above was fine on the day we looked, which is the only thing a one-off check can ever establish.

Method: 624 sites, checked 27 August 2026. Signal one, robots.txt, parsed to RFC 9309. Signal two, a paired browser/crawler probe of the homepage. Every finding reported here was re-tested with the crawler request first, six seconds apart, across GPTBot, ClaudeBot and PerplexityBot. Sites behind a bot-verifying CDN are reported as unverifiable and are excluded from the block counts. We intend to re-run this quarterly; the trend matters more than any single number.