AI search October 2, 2026
AI crawlers and robots.txt: what to allow and what to block
OpenAI, Anthropic and Perplexity each run several bots: for training, for search and for user requests. Keep them separate in robots.txt.
One robots.txt line does not settle the AI question, because the same company runs several bots with different purposes. Confusion starts when you block everything at once and lose visibility too.
Three roles: training, search, user request
- Training: the bot collects content to train models (GPTBot, ClaudeBot)
- Search: the bot indexes pages as sources for answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
- User request: the bot reads a page when a person asks an assistant about it (ChatGPT-User, Claude-User, Perplexity-User)
What each service says itself
According to OpenAI’s documentation, disallowing OAI-SearchBot means a page will not appear in ChatGPT search answers, though it can still show as a navigational link. A change takes about 24 hours to apply. Disallowing GPTBot signals that content must not be used for training. ChatGPT-User runs on a user’s request, and OpenAI notes that robots.txt rules may not apply there.
Anthropic describes three bots: ClaudeBot (training), Claude-User (user query) and Claude-SearchBot (search quality). Its bots honour robots.txt, do not try to bypass CAPTCHAs and support the non-standard Crawl-delay directive.
Perplexity splits its bots in two: PerplexityBot surfaces sites in search results and is not used for training models, while Perplexity-User acts on a user’s request and generally ignores robots.txt. Changes take up to 24 hours to apply.
An example robots.txt policy
Example: you want to be visible in ChatGPT and Claude but do not want to hand over training data. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot (Allow: /) and disallow GPTBot and ClaudeBot (Disallow: /). Manage Google-Extended separately: it concerns Google’s AI training and other systems, not ordinary search.
What to watch
- Do not block Googlebot: Google’s AI Overviews and AI Mode use the regular Search index
- User-triggered bots may not follow robots.txt, so protect sensitive content with a password or header, not robots.txt
- IP blocking is not reliable; use the IP lists the companies publish together with the User-Agent
- A CDN or WAF rule can block a bot even when robots.txt allows it, so test with a real request
How to decide
- Decide whether your content is sold, licensed, or public marketing material
- Marketing material: allow search bots, because visibility is the goal
- Paid or licensed data: disallow training bots and consider access control
- Write one rule per bot and add a comment explaining why
- Check server logs monthly for who really visits, and update the list
What to check each quarter
- Whether the services’ documentation has changed the list of bots
- Whether robots.txt matches your current decision
- Whether the server log has bots you did not expect
- Whether WAF or CDN rules block bots you allowed
Services add and rename bots, so a regular check keeps your decision and reality aligned.
A practical note
Keep robots.txt under version control and comment every line with why it is there. In a year it is your most valuable document when someone asks why a certain bot is allowed or disallowed.
Three questions
Does disallowing GPTBot remove me from ChatGPT search?
No. GPTBot is for training, while OAI-SearchBot handles ChatGPT search. Each has its own rule and they can be configured differently.
How do I verify a bot really is OpenAI, Anthropic or Perplexity?
The companies publish IP lists: OpenAI at openai.com/searchbot.json and openai.com/gptbot.json, Anthropic at claude.com/crawling/bots.json and Perplexity at perplexity.com/perplexitybot.json. Compare them with your server log, not just the User-Agent, because the latter can be forged.
What if the WAF blocks a bot?
Add an exception for the combination of User-Agent and the official IP list, and refresh the list regularly. Perplexity recommends the same approach.
Summary
Decide the goal, write a separate rule for each bot and check the log for who really visits. Changes can take up to 24 hours to take effect.
