robots.txt for AI Crawlers: Which to Allow and Which to Block

Search crawlers and training crawlers are different bots

Most AI companies now run at least two kinds of bot. One collects pages that may be used to train models. The other fetches pages to show and cite in search answers. A third kind fetches a page only when a person asks the assistant to open it.

That split matters, because the most common mistake in 2026 is a robots.txt or firewall rule that blocks "all AI bots" and quietly removes a site from ChatGPT search or Perplexity answers, when the owner only meant to opt out of training.

Sources, checked October 2026: OpenAI, Anthropic, Perplexity, Google.

What Google says about AI Overviews

Google says there are no additional requirements to appear in AI Overviews or AI Mode, and no special schema or AI text files are needed. A page has to be indexed and eligible to show with a snippet (Google: AI features and your website). So the crawler that matters for AI Overviews is plain Googlebot.

Google-Extended is different. Google says it controls whether content may be used to train future Gemini models and for grounding in Gemini apps, and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Blocking it is a training decision, not a visibility one.

If you want to limit how much of a page Google shows in any snippet, including AI features, the tools are nosnippet, data-nosnippet and max-snippet. Use them on purpose, not by accident: a leftover max-snippet:0 makes a page hard to cite.

Three copy-paste robots.txt setups

Setup A: be visible everywhere, allow training too. This is what you get with no AI rules at all. Make sure nothing else blocks these bots:

User-agent: *
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Setup B: be visible in AI search, opt out of training. The most common choice for businesses that want citations but not training use.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot fall under User-agent: * and stay allowed.

Setup C: block one folder from AI search only. For example, paid member content:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /members/

Remember that Google (and crawlers that follow the robots.txt standard) obey only the most specific group that names them. If you add a group for OAI-SearchBot, it ignores your User-agent: * rules, so repeat any Disallow lines you still need in that group.

Check your firewall, not just robots.txt

robots.txt is a request. Firewalls and CDNs are a wall. Many hosts and CDNs now offer one-click "block AI bots" settings, and some turn them on by default. Those settings may stop search crawlers as well as training crawlers.

1. Open your CDN or security plugin and look for bot, AI or crawler settings.
2. Check which bots the rule lists. If OAI-SearchBot, Claude-SearchBot or PerplexityBot are on it and you want citations, allow them.
3. Check your server logs for these user agents and their status codes. 403 responses mean the wall is up.

How to test it

• Fetch your robots.txt and read it top to bottom. Look for User-agent: * with Disallow: /, which blocks everything.
• Use a robots.txt parser (several free ones exist online) to test a URL against each user agent token.
• Fetch a key page with curl and confirm the main text is in the raw HTML. Many AI crawlers do not run JavaScript, so a page that only renders in the browser can look empty to them.

Worked example (fictional site). brightsmile-dental.com turned on a CDN setting called "Block AI scrapers" in March. Its robots.txt still allowed every bot, so the owner assumed nothing had changed. Server logs showed OAI-SearchBot and PerplexityBot getting 403 on every request since that day. The fix was to keep the training bots blocked and allow the two search bots in the CDN rule.