GPTBot crawls content that may be used to train OpenAI models, while OAI-SearchBot indexes pages specifically for ChatGPT Search links and summaries. PerplexityBot indexes web pages to surface cited answers in Perplexity. Configuring each user agent individually in robots.txt prevents accidental search blackouts.
Enter your website address to check crawler access (GPTBot, ClaudeBot), server-rendered HTML, and JSON-LD schemas.
Why training opt-outs often break AI search visibility
AI providers separate crawlers by operational role because site owners have different policies for model training versus real-time search citations. When a robots.txt rule uses an overbroad disallow or blocks every AI user agent indiscriminately, the website disappears from live answer-engine citations even when the owner only meant to opt out of pre-training datasets.
How the major AI user agents differ
Before editing robots.txt, map each crawler token to its actual function on the web.
- GPTBot (OpenAI): Crawls public pages that may be used to improve and train OpenAI foundation models.
- OAI-SearchBot (OpenAI): Discovers and indexes websites specifically to appear as search results and citations in ChatGPT Search.
- ChatGPT-User (OpenAI): Fetches a specific URL on demand when a ChatGPT user asks the assistant to read or act on a page.
- PerplexityBot (Perplexity): Discovers and indexes web content to power Perplexity’s real-time search answers and source links.
- ClaudeBot (Anthropic): Web crawler documented by Anthropic; review Anthropic’s current guidance before setting crawl policies.
Recommended robots.txt configurations by policy
Choose the ruleset that matches your business intent, and always keep private dashboards or customer endpoints behind server authentication rather than relying on robots.txt alone.
# Option A: Allow AI search citations, disallow OpenAI model training
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
# Protect private routes for all crawlers
User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://your-site.example/sitemap.xmlCheck beyond robots.txt: firewalls and rate limits
Permitting OAI-SearchBot or PerplexityBot in robots.txt does not guarantee retrieval if your CDN, WAF, or host rate limiter blocks automated requests with HTTP 403 or a JavaScript challenge page. Use SiteReadyFor.SI’s free AI bot checker to inspect robots.txt directives per URL, and compare server logs when diagnosing dropped crawls.