ChatGPT Search discovers websites through OpenAI’s dedicated search crawler (OAI-SearchBot), real-time user-triggered fetches (ChatGPT-User), and external search indexes. Allowing OAI-SearchBot in robots.txt, serving substantive text in initial HTML, and publishing an accurate sitemap keep your pages discoverable.
Enter your website address to check crawler access (GPTBot, ClaudeBot), server-rendered HTML, and JSON-LD schemas.
Three retrieval paths behind ChatGPT Search
When someone asks ChatGPT a web-grounded question, it does not rely on model memory alone. Discovery combines existing search index partnerships with OpenAI’s own search crawler (OAI-SearchBot) and live on-demand page retrieval (ChatGPT-User) when a user shares a URL or triggers a deep lookup. Separating these paths explains why a site can appear in traditional search yet fail live ChatGPT retrieval if its host blocks OpenAI’s search user agents.
Do not block OAI-SearchBot when opting out of training
Many teams add a broad GPTBot disallow rule to opt out of foundation model training and assume search visibility is unaffected. In OpenAI’s crawler documentation, GPTBot handles training crawls whereas OAI-SearchBot indexes websites specifically for surfacing links and answers in ChatGPT Search. Keep your robots.txt rules explicit per bot so your intended search policy matches your training policy.
# Allow ChatGPT Search discovery while keeping GPTBot training separate
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
Sitemap: https://your-site.example/sitemap.xmlWhy raw server HTML, sitemaps, and canonicals decide ingestion
Search-retrieval crawlers fetch pages quickly at scale and rarely wait through multi-stage client-side JavaScript hydration. If headings, product specifications, or direct answers only render after React executes in a browser, the crawler extracts an empty shell and moves to another source.
- Verify your public pages return HTTP 200 with full headings and body copy in initial HTML using curl.
- List canonical public URLs in /sitemap.xml and exclude redirects, drafts, or soft-404 routes.
- Place a concise factual summary directly under the H1 so RAG chunking extracts the answer cleanly.
- Check CDN and WAF logs for 403 or challenge pages served to OAI-SearchBot or ChatGPT-User.
Check ChatGPT Search readiness with SiteReadyFor.SI
SiteReadyFor.SI inspects your robots.txt rules for OAI-SearchBot, ChatGPT-User, and GPTBot, checks whether important text exists in initial server HTML, validates your XML sitemap, and parses JSON-LD. It verifies the technical retrieval layer your site controls; it does not query live ChatGPT prompts or guarantee brand citations.