Every major AI crawler, who runs it, what it does with your pages, and the robots.txt rule to block it. Last reviewed October 24, 2026 against each company's own documentation.
We track 22 AI crawler names that you can use in robots.txt. They fall into three kinds:
Blocking one kind doesn't block the others. To stop AI training but still be cited in AI answers, block only the training crawlers.
Collect pages to train AI models.
Crawls pages that may be used to train OpenAI's models, including the models behind ChatGPT.
| If you block it | Your pages shouldn't be used to train OpenAI models. Doesn't affect ChatGPT search, which uses OAI-SearchBot. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | developers.openai.com |
Collects public web pages that may be used to train Anthropic's Claude models.
| If you block it | Your future pages should be left out of Claude training data. Doesn't affect Claude-SearchBot or Claude-User. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | support.claude.com |
Not a separate crawler. This token tells Google whether pages Googlebot has crawled may be used to train Gemini models and to ground Gemini answers.
| If you block it | Your pages shouldn't be used for Gemini training or grounding. Google says it doesn't affect Google Search or ranking. AI Overviews are part of Google Search, so this token doesn't control them. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | developers.google.com |
Not a separate crawler. This token tells Apple whether pages crawled by Applebot may be used to train Apple's generative AI models.
| If you block it | Opts your pages out of Apple's AI model training. They can still appear in Siri, Spotlight and Safari search. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | support.apple.com |
Crawls pages to train Meta's AI models and to improve Meta products by indexing content directly.
| If you block it | Your pages shouldn't be used to train Meta's AI models. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | developers.facebook.com |
Builds Common Crawl, a free public archive of the web. The archive is one of the most widely used sources of AI training data.
| If you block it | Future crawls leave your pages out of the archive that many AI companies download. Pages in past archives stay there. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | commoncrawl.org |
ByteDance's crawler (the company behind TikTok). ByteDance hasn't published documentation for it. It's widely reported to collect data for ByteDance's AI models.
| If you block it | Asks ByteDance not to crawl your site. Independent reports say it doesn't always comply, so consider blocking it at your firewall or CDN too. |
|---|---|
| Follows robots.txt | Reportedly not always |
| Official docs | None published |
Amazon's general crawler. Amazon says what it collects improves its products and may be used to train Amazon AI models.
| If you block it | Your pages shouldn't be used to train Amazon AI models. Alexa answers use Amzn-SearchBot and Amzn-User, which you control separately. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | developer.amazon.com |
Crawls pages to build datasets for training Mistral's AI models. Not used for search or live answers.
| If you block it | Your pages shouldn't be used to train Mistral models. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | docs.mistral.ai |
Index pages so AI assistants can find, cite and link to them.
Indexes pages so they can appear, with links, in ChatGPT search answers.
| If you block it | Your pages won't appear in ChatGPT search answers, though they can still show up as plain navigational links. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | developers.openai.com |
Indexes pages to improve the quality of Claude's search results.
| If you block it | Your pages aren't indexed for Claude's search, so Claude may find and cite them less often. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | support.claude.com |
Indexes pages so they can appear, with links, in Perplexity answers. Perplexity says it isn't used to collect data for training AI models.
| If you block it | Your pages won't be indexed for Perplexity search, though Perplexity-User may still open them when a user asks. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | docs.perplexity.ai |
Indexes pages to improve Meta AI search results, so Meta AI can cite and link to sites in its answers.
| If you block it | Meta AI is less likely to cite and link to your pages. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | developers.facebook.com |
Indexes pages for search features in Amazon products, such as Alexa answers. Amazon says it isn't used for AI model training. If your robots.txt doesn't name it, Amazon says it follows the rules you set for other search crawlers.
| If you block it | Your pages are less likely to appear in Alexa and other Amazon search features. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | developer.amazon.com |
Indexes pages for Mistral's search, which helps its AI assistant answer questions. Mistral says indexed pages aren't used for training.
| If you block it | Your pages are less likely to appear in Mistral's AI answers. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | docs.mistral.ai |
Fetches pages for DuckDuckGo's AI-assisted answers, which show their sources. DuckDuckGo says the data isn't used to train AI models.
| If you block it | Your pages won't be used as sources in DuckDuckGo's AI answers. DuckDuckGo says changes take effect after 72 hours. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | duckduckgo.com |
Fetch a page when someone asks an AI assistant about it.
Visits a page when a ChatGPT user or custom GPT asks about it. Not used for automatic crawling.
| If you block it | Limited effect: OpenAI says robots.txt rules may not apply to visits a user started. |
|---|---|
| Follows robots.txt | May ignore it for user requests |
| Official docs | developers.openai.com |
Fetches a page when a Claude user's question needs it.
| If you block it | Claude can't open your pages for users, which may reduce how often your site shows up in answers. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | support.claude.com |
Visits a page when a Perplexity user asks a question that needs it, and may link to it in the answer. Not used for training.
| If you block it | Limited effect: Perplexity says this fetcher generally ignores robots.txt because a user requested the page. |
|---|---|
| Follows robots.txt | May ignore it for user requests |
| Official docs | docs.perplexity.ai |
Fetches individual links at a user's request for Meta's AI assistant and agent features.
| If you block it | Limited effect: Meta says it may bypass robots.txt because a user requested the page. |
|---|---|
| Follows robots.txt | May ignore it for user requests |
| Official docs | developers.facebook.com |
Fetches live pages for user requests, such as Alexa questions that need up-to-date answers. Amazon says it isn't used for AI training.
| If you block it | Partial effect: Amazon says it may not follow all robots.txt rules for requests a user started. |
|---|---|
| Follows robots.txt | May ignore it for user requests |
| Official docs | developer.amazon.com |
Visits a page when a user asks Mistral's AI assistant a question, and links to it as a source. Not used for training.
| If you block it | Mistral's assistant shouldn't open your pages for users. |
|---|---|
| Follows robots.txt | Yes |
| Official docs | docs.mistral.ai |
These still appear in many robots.txt files, but the companies no longer list them. Blocking only these may not block anything today.
An older Anthropic token that no longer appears in Anthropic's crawler documentation.
Use instead: ClaudeBot
An older Anthropic token that no longer appears in Anthropic's crawler documentation.
Use instead: ClaudeBot and Claude-User
An older Meta crawler that no longer appears in Meta's crawler documentation.
Use instead: meta-externalagent
No. GPTBot is only for training. ChatGPT search uses OAI-SearchBot, and pages a user asks ChatGPT to open use ChatGPT-User. OpenAI treats each one separately, and most other companies split their crawlers the same way.
Not with Google-Extended. It only controls Gemini training and grounding. AI Overviews are part of Google Search and use Googlebot, so blocking them in robots.txt would also remove you from search results. Google's snippet controls, such as the nosnippet tag, limit what can be shown instead.
robots.txt is a voluntary standard (RFC 9309), not a technical block. The large AI companies say their crawlers follow it, but several say their user-triggered fetchers may not, because a person asked for the page. To stop crawlers that ignore it, block them at your server, firewall or CDN.
At the root of each host, such as https://example.com/robots.txt. Subdomains like blog.example.com need their own file. Most crawlers notice changes within about a day.