The before-picture window has closed
The offer on this page was to record how AI crawlers saw your client sites before Cloudflare’s 15 September change. That cannot be done now. Not by us, not by anyone — a before-picture cannot be taken afterwards, which was the whole reason for the deadline.
What we measured. 1,046 sites, ten identities each, four runs across two weeks — published in full, including the drift control and what the sample is and is not. Read the study.
Where your own site stands today. The checker still runs on anything, with no account — check a site. It tells you today’s state, not what changed. Nothing can tell you what changed.
If this date was useful to anyone, it was useful as a reminder that crawler access leaves no record. The free plan watches one site, free, no card. The point of starting now is not this change — it is that the next one will also arrive without warning, and next time there will be a before.
First, the part everyone is getting wrong
| Question | Answer |
|---|---|
| When | 15 September 2026 |
| Who is affected | New Cloudflare customers, new sites added by existing customers, and all existing free-tier customers |
| Which pages | Only pages that display ads |
| Which crawlers | Training and Agent categories. Search crawlers are not affected |
| Can you override it | Yes. Security → Bots. About twenty minutes |
Why the date mattered
Crawler access leaves no trace in analytics. There is no log. And robots.txt only tells you what is declared, not what your CDN or your host's firewall actually does to a request. So since 15 September, when somebody asks whether anything changed for their site, the honest answer available to almost everybody is a shrug.
That was fixable only in one direction. The measurement had to happen first.
What the accounts that took it got
Before the 15th
19 crawlers and opt-out rules tested against every site you add, plus robots.txt, JavaScript rendering and llms.txt. Stored with a timestamp.
On the 15th
Everything is re-checked on the day itself and again the morning after, so a change lands inside the window rather than up to a day later.
On the 16th
Every site you added has its own before-and-after in its history, timestamped either side of the change. The aggregate across all 1,046 sites we measured is published at /research/september-15.
What it costs now
The free plan watches one site, free, no card. Until 19 September it took 25; that offer has ended and adding a second site on the free plan is refused. The plans are $19 for five sites and $49 for twenty-five.
Nothing auto-charged and nothing was deleted — accounts that took the offer keep being checked on the sites they added, and the history stays on the account.
Do I have to verify my domains?
No. Sites you add are checked from the outside, exactly the way an AI crawler would visit — the same request anyone can already run on the free checker without an account. Nothing to install, no DNS record, no file upload.
Verification is only needed for the things that reach into a site or speak for it: alert emails about a domain, and verified-visit tracking. Those stay gated. Baselining does not, and neither does the report — every site you add gets one, whether or not you verify it.
- Unverified — checks, full history including the 15th and 16th, and a client-ready report for every site.
- Verified — all of that, plus alert emails and verified-visit tracking.
It is one DNS record, file or meta tag per domain, it takes a minute, and you can do it later for the sites where you want the alerts.