Guide
Every AI crawler, and whether you should let it in
There are far more of these than most people realise, they do noticeably different jobs, and two of the most commonly blocked are not crawlers at all.
The three kinds
Lumping them together is the mistake that causes most accidental damage.
- Answer crawlers build the index an AI cites when it answers a question. Blocking one removes you from that assistant's answers. This is almost always the wrong thing to block.
- User-triggered fetches happen because a real person asked the AI about your page, right now. Blocking these breaks something a potential customer is actively trying to do.
- Training collectors gather text to train future models. This is the one there is a genuine argument about — publishers with valuable archives may reasonably say no.
The most common expensive mistake. Somebody decides "we do not want AI training on our content", pastes a block list from a forum, and takes out the answer crawlers and user fetches along with the training ones. The site quietly disappears from AI answers while the training data it was worried about is already in older models.
The full list
| Name | Operator | What it does | Cost of blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content to train future models. | Indirect. Affects what ChatGPT knows about you in general. |
| OAI-SearchBot | OpenAI | Builds the index ChatGPT search draws answers from. | High. Block this and you cannot be cited in ChatGPT search. |
| ChatGPT-User | OpenAI | Fetches a page live because a person asked ChatGPT about it. | High. This is someone actively researching you right now. |
| ClaudeBot | Anthropic | Collects content for training. | Indirect. |
| Claude-SearchBot | Anthropic | Indexes pages for Claude to cite in answers. | High. |
| Claude-User | Anthropic | Fetches a page because a Claude user asked about it. | High. |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers. | High. |
| Perplexity-User | Perplexity | Live fetch driven by a person asking a question. | High. |
| Googlebot | The ordinary Google crawler. Also feeds AI Overviews. | Critical. Blocking it removes you from Google entirely. | |
| Google-Extended | Not a crawler. A permission setting for Gemini training. | Indirect. Blocking does not affect Google Search. | |
| bingbot | Microsoft | Bing index, which also feeds Copilot. | High. |
| Applebot | Apple | Siri and Spotlight. | Medium. |
| Applebot-Extended | Apple | Permission setting for Apple AI training. Not a crawler. | Indirect. |
| meta-externalagent | Meta | Collects content for Meta AI. | Medium. |
| Amazonbot | Amazon | Feeds Alexa and Rufus answers. | Medium. |
| DuckAssistBot | DuckDuckGo | Powers DuckDuckGo AI answers. | Medium. |
| MistralAI-User | Mistral | Live fetch for Le Chat users. | Low to medium. |
| CCBot | Common Crawl | Open web archive many models train on. | Indirect but wide reaching. |
| Bytespider | ByteDance | Collects content for ByteDance models. | Low. |
How to allow all of them
The simplest correct robots.txt for most businesses is short:
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
If you want to refuse training but stay visible in answers, block only the training collectors and leave the rest alone:
# Refuse training, stay citable in AI answers
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Everything else stays welcome
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Everything else stays welcome
User-agent: *
Allow: /
robots.txt is only half the story
A perfect robots.txt does not help if your firewall, CDN or security plugin is refusing crawler traffic before it ever reaches your server. That mismatch is invisible from the outside unless you go looking for it, which is why we built SeenSure.