Most websites fail AI readiness audits for four recurring technical reasons: an unintended robots.txt or firewall block on search crawlers, client-rendered JavaScript hiding primary copy from initial HTML, syntax errors in JSON-LD, and catch-all routes returning HTML instead of /sitemap.xml or /llms.txt. Fixing and verifying each signal in order resolves the root retrieval bottlenecks.
Enter your website address to check crawler access (GPTBot, ClaudeBot), server-rendered HTML, and JSON-LD schemas.
Baseline audit findings: why a working site failed retrieval
Consider a representative B2B SaaS and product storefront walkthrough modeled on common production failures (illustrative composite example; real customer reports remain private). In a browser, the homepage, pricing, and docs look complete. When audited with an HTTP-first scanner, four concrete failures appear: discovery finds only 1 URL, OAI-SearchBot is blocked, raw HTML holds an empty React root shell, and JSON-LD throws a parsing error.
Problem 1 & fix: Blocked search bot and missing sitemap URLs
Inspection showed /robots.txt disallowed all bots except Googlebot, while /sitemap.xml returned the React app shell HTML because the catch-all SPA rewrite ran before static files. Moving sitemap generation ahead of the app router and allowing search bots restored clean XML discovery.
# Before: catch-all rewrite served index.html at /sitemap.xml
# and robots.txt blocked OAI-SearchBot & PerplexityBot
# After: explicit static XML + search crawler rules
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://your-site.example/sitemap.xmlProblem 2 & fix: Empty React root container in initial HTML
Running curl against /pricing returned only a 12-line HTML shell with <div id="root"></div> and a JavaScript bundle. Any crawler skipping full browser execution saw zero pricing, features, or headings. Enabling server-side rendering (SSR) or build-time prerendering for public marketing and pricing routes placed the complete H1, feature list, and FAQ text directly in the HTTP 200 response.
Problem 3 & fix: Broken JSON-LD comma and /llms.txt HTML fallback
A trailing comma inside the template’s Organization JSON-LD stopped parsers from reading brand details, and /llms.txt served the homepage HTML instead of plain Markdown. Removing the trailing comma, aligning JSON-LD fields with visible page copy, and publishing a static text/plain file at /llms.txt resolved both checks.
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"name": "Acme Analytics",
"applicationCategory": "BusinessApplication",
"offers": {
"@type": "Offer",
"price": "29",
"priceCurrency": "USD"
}
}Verification workflow after deploying fixes
Never rely on local development checks alone; verify every change against the live production origin before closing the audit.
- Run curl -sL on each fixed route to confirm H1, pricing, and core links appear in raw HTML.
- Fetch /sitemap.xml and /llms.txt directly and verify XML or plain-text Markdown instead of HTML.
- Rerun page discovery and audit the same page scope in SiteReadyFor.SI to confirm issues clear cleanly.
- Track traffic and AI search referrals separately in analytics rather than treating a score as a ranking guarantee.