Most sites optimize for Googlebot and forget that ChatGPT, Perplexity, Claude, and Google's AI Overviews use different crawlers. If your robots.txt blocks OAI-SearchBot, ChatGPT search cannot show your pages. If PerplexityBot is blocked, you will not appear in Perplexity citations. Each company also runs a separate training bot (GPTBot, ClaudeBot, Google-Extended): many site owners block those on purpose, and a broad rule written for them often catches the search bots by accident.
Our checker fetches your URL plus your robots.txt and leads with the result that actually matters: a 6-bot matrix. The search bots decide whether you can be cited: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude web search) and PerplexityBot (Perplexity). The training bots are your choice: GPTBot (OpenAI model training), ClaudeBot (Anthropic model training) and Google-Extended (Gemini apps, not Google Search or AI Overviews). When a bot is blocked, its tile shows the exact Disallow line from your robots.txt and a one-line fix you can copy-paste straight into your config. When all are allowed, you get a green celebration card so you can move on. Below the matrix, the tool runs a secondary technical crawlability pass covering HTML cleanliness (content-to-code ratio), heading hierarchy, list and table usage, and average paragraph length, because even when bots can reach your page, messy markup makes it hard to parse.
For SEOs and Site Owners
- Catch accidental AI bot blocks in 5 seconds (most common: a blanket Disallow rule that catches the search bots along with the training bots)
- See which AI surfaces will and will not surface your content
- Identify the top technical fixes that improve how AI parsers extract your content
For Agencies
- Run this on every new client URL during onboarding. The bot-block check alone has caught critical misconfigurations in real audits.
- Track AI crawlability as a separate metric alongside traditional Googlebot crawlability
Why Use This Tool
- Per-bot blocking-rule matrix as the hero: Each blocked bot shows the exact
Disallow line to remove, not just a red/green chip. Copy-paste the rule out of your robots.txt and you're fixed.
- Six bots, all real: We check the user agents OpenAI, Anthropic, Perplexity and Google document, and label each one as search or training. Not made-up user-agents.
- Free, instant, no signup: Paste URL, solve captcha, get answer.
