Letting AI Crawlers Read Your Site
An AI assistant can only quote a page its crawler is allowed to read. Your robots.txt file decides that, and it is the most common reason a site is missing from AI answers - often because of a rule written years ago for a different reason.
Search Crawlers and Training Crawlers
The report checks 16 crawlers, in two groups.
Search and answer crawlers fetch pages to answer a question or cite a source right now. If one is blocked, that assistant cannot use your page at all, so each block costs heavily in your score.
- OAI-SearchBot and ChatGPT-User - ChatGPT search, and pages ChatGPT visits when someone asks it to
- Claude-SearchBot and Claude-User - Claude search, and pages Claude visits on request
- PerplexityBot and Perplexity-User - Perplexity search, and pages it visits on request
- Googlebot - Google Search, including AI Overviews and AI Mode
- Bingbot - Bing, which powers Copilot
- Applebot - Siri and Spotlight
- DuckAssistBot - DuckDuckGo's DuckAssist answers
Training crawlers collect text to train future models. Blocking them is a legitimate choice, so the report only ever marks it as Improve.
- GPTBot - OpenAI model training
- ClaudeBot - Anthropic model training
- Google-Extended - Gemini training and grounding
- Applebot-Extended - Apple model training
- Meta-ExternalAgent - Meta AI training
- CCBot - Common Crawl, an open dataset many models are trained on
Allowing AI Search Crawlers
robots.txt lives at the root of your site, for example example.com/robots.txt. This example lets the AI search crawlers in while keeping an admin area private:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlA Named Crawler Ignores the * Group
A crawler that has its own group in robots.txt follows only that group and ignores the rules under "User-agent: *". When you add a group for a crawler, copy into it any Disallow rules you still want it to obey.
Other Things That Hide a Page
- A noindex robots meta tag or X-Robots-Tag header keeps the page out of search results, and the AI answers built on them
- nosnippet stops Google quoting the page, including in AI Overviews
- Some hosting and bot-protection services block AI crawlers at the firewall, separately from robots.txt. We cannot see those settings, but if our own crawler is turned away, the report's "Page reachable by crawlers" check says so
Content in the HTML
Many AI crawlers do not run JavaScript. If your page builds its text in the browser, the "Readable without JavaScript" check will flag it - render the main content on the server instead.