Robots.txt became an official internet standard in 2022 as RFC 9309, decades after Martijn Koster first proposed it in 1994 (RFC Editor). The important thing it now lets you do: name individual AI crawlers and give each one a different answer, because the major AI companies run separate bots for separate jobs.
AI crawlers do three different things: index your pages for search, fetch a page live when a user asks, and collect content for model training. Robots.txt can allow the first two while blocking the third, using per-bot rules. That posture is “found, not trained”: cited in AI answers, kept out of the training corpus. It is a deliberate choice, and most sites make it by accident.
Why does one robots.txt need three answers?
Because “AI bot” is not one thing. OpenAI, Anthropic, and Perplexity each publish separate user-agents, and their own documentation states that blocking one does not block the others (OpenAI, Anthropic). A search bot that surfaces you in ChatGPT’s search results is a different process from the training crawler that vacuums text for the next model. Block them with one lazy Disallow and you throw out the citation with the training.
Here is the current roster, verified against each provider’s official crawler documentation:
| User-agent | Role | What blocking it costs you |
|---|---|---|
OAI-SearchBot |
Search index (ChatGPT search) | You disappear from ChatGPT’s search citations |
ChatGPT-User |
Live fetch (user-initiated in ChatGPT) | ChatGPT cannot open your page when a user asks |
GPTBot |
Training (OpenAI foundation models) | Excluded from OpenAI training data |
Claude-SearchBot |
Search index (Claude search) | Reduced visibility in Claude’s search answers |
Claude-User |
Live fetch (user-initiated in Claude) | Claude cannot fetch your page for a user query |
ClaudeBot |
Training (Anthropic models) | Excluded from Anthropic training data |
PerplexityBot |
Search index (Perplexity citations) | Not indexed or cited by Perplexity |
Perplexity-User |
Live fetch (user-initiated) | Little: it may fetch a user-supplied URL regardless |
Google-Extended |
Training token (Gemini, Vertex AI) | Excluded from Gemini training, no effect on Search rank |
Bingbot |
Search index (Bing, and ChatGPT upstream) | You vanish from Bing and, downstream, from ChatGPT |
CCBot |
Open training dataset (Common Crawl) | Near zero for AI citation: safe to block |
Sources for the tokens and roles: OpenAI’s bot documentation, Anthropic’s crawler help article, Perplexity’s crawler docs, and Google’s Google-Extended token.
What does “found, not trained” look like in the file?
You allow the search and live-fetch bots, and you disallow the training bots. The training blocks that carry no search cost are Google-Extended and CCBot: Google confirmed Google-Extended is not a ranking signal and does not affect Search inclusion, and CCBot only feeds an open dataset that your AI-search visibility does not depend on (Common Crawl).
The one that is not a clean win is GPTBot. It is labeled training, and blocking it keeps you out of the training corpus, but see the GPTBot trap before you reach for the block: the search bot you actually need is a different token, and people confuse the two constantly.
Bingbot deserves its own line. It is not marketed as an AI crawler, but ChatGPT’s search leans heavily on Bing’s index. Block Bingbot to keep out scrapers and you can quietly delete yourself from ChatGPT at the same time. It is the friendly-fire incident of AI visibility.
How do we check this in practice?
When we scan a site, the robots.txt check does not just look for a blanket block. It reads each of these eleven tokens and reports which are allowed, which are disallowed, and whether the resulting posture is coherent. The common failure is not aggression, it is accident: a Disallow: / under User-agent: * that the site owner assumed applied only to scrapers, silently blocking OAI-SearchBot and Claude-SearchBot along with them. The second most common: robots.txt served as an HTML page with a 200 status, which some crawlers treat as no rules and others treat as garbage.
The practical next step: open your robots.txt, search it for each of the eleven tokens above, and decide the search and fetch bots on purpose instead of by default. If you want the posture graded for you, run a free scan and see which AI crawlers your site currently allows, blocks, or blocks without meaning to.