Three signals, ranked by how much we trust them
Most checkers give you a confident yes or no. Some of those answers are guesses. Here is exactly what we can prove and what we cannot.
Signal 1 — what your robots.txt declares
Every AI crawler reads a file at yoursite.com/robots.txt before it visits, and obeys what it finds. If that file says a crawler is not welcome, the crawler stays away. There is no ambiguity here, so we treat this as fact.
This one check catches a surprising share of real problems: templates copied from another site, a line added years ago to stop scrapers, or a security plugin that rewrote the file during an update.
User-agent: GPTBot
Disallow: /
Two lines. Nobody remembers adding them. ChatGPT has not read the site since.
Signal 2 — what a live crawler request actually gets
robots.txt is only a request. A server can still refuse a crawler outright. So we ask for your page twice, at the same moment, from the same machine — once identifying as an ordinary browser, once as the crawler. Then we compare.
| What we see | What it means |
|---|---|
| Both get the page | You are fine. |
| Browser gets it, crawler is refused | A real block. We tell you where and how to lift it. |
| Both get a much smaller page | Content is hidden behind JavaScript. |
| Neither gets it | Your site is down. Not an AI problem — we say so rather than raising a false alarm. |
Signal 3 — verified crawler visits
Here is the honest limitation nobody else mentions. Cloudflare, Fastly, Akamai and Imperva do not simply believe a request that says "I am GPTBot". They check the visitor's IP address against the crawler operator's published list, and increasingly a cryptographic signature too. No outside tool can imitate that — ours included.
So when your site sits behind one of those, a refusal might mean your settings block the crawler, or it might just mean the CDN correctly spotted that we are not the real thing. We report that as "cannot confirm from outside" rather than calling it a block and sending you chasing a problem that may not exist.
To settle it properly, Pro plans can turn on verified-visit tracking. We then watch for genuine crawler visits and check their IP addresses against the official published lists. That gives you something no outside test can:
What else we check
- JavaScript dependency. Most AI crawlers do not run JavaScript. If your content only appears after the browser does work, they see an empty page.
- llms.txt. Optional, but a broken one is worse than none — it hands crawlers a map to pages that no longer exist. We check every link.
- Page basics. Title, description, headings, structured data, sitemap. These decide whether an AI describes you accurately or invents something.
How often
Every six hours on paid plans, daily on the free plan. We only email you when something changes — a break, or a recovery. Silence means everything is fine.