19 days to go

Cloudflare changes its AI crawler defaults on 15 September 2026

If you or your clients sit on Cloudflare's free tier, some pages may stop being readable by AI without anyone touching a setting. Here is the accurate version, without the scaremongering.

What actually changes

Cloudflare is pressuring AI companies to separate crawlers used for search from those used for training and agents. To apply that pressure, it is changing what happens by default when a site owner has not chosen anything.

QuestionAnswer
When15 September 2026
Who is affectedNew Cloudflare customers, new sites added by existing customers, and all existing free-tier customers
Which pagesOnly pages that display ads
Which crawlersTraining and Agent categories. Search crawlers are not affected
Can you override itYes. It is a setting you control
Read that "only pages that display ads" line carefully. This is not the whole web going dark, and plenty of coverage has implied it is. If your sites carry no advertising, this specific change does not affect them.

Why it still matters

Because of who sits on the free tier: small business sites, and an enormous number of client sites that agencies put behind Cloudflare years ago and never looked at again. Nobody who set those up is watching a policy page.

There are two ways this bites quietly. You may end up blocking training crawlers when you would rather have been paid for that content. Or you block agent traffic, which means a real person asking an AI to look at your site gets nothing back. The second is the one that actually costs you a customer.

What to do, in about twenty minutes

  1. Log into Cloudflare and open Security → Bots for each domain.
  2. Find the AI crawler controls. There are three categories: Search, Agent, Training. Decide each one deliberately instead of inheriting a default.
  3. Check your robots.txt separately. Cloudflare's setting and your robots.txt are two different gates, and both have to be open.
  4. If you have valuable content, look at Cloudflare's Pay Per Use before switching training back on. You may be able to get paid for it.

Then check it stayed that way

This is the part people skip. Setting it correctly once does not keep it correct. Defaults change, plugins update, sites get migrated, and staff leave. The reason SeenSure exists is that nothing tells you when a setting you chose last year has quietly stopped applying.

Check a site right now Watch it continuously

Sources: Cloudflare's July 2026 announcement as reported by TechCrunch and others. We will update this page if the scope changes before the date.