Let ChatGPT, Claude and Perplexity Read Your Site: AI Crawlers and robots.txt
Updated
Which AI Bots Matter#
| Kind of bot | Examples | What it means for you |
|---|---|---|
| Answer-time fetches | ChatGPT-User, Claude-User, Perplexity-User, Amzn-User (Alexa), Meta-ExternalFetcher (Meta AI) | The bot read your page to write an answer someone is reading now. These visits go with citations. |
| AI search indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot (Siri, Spotlight), Amzn-SearchBot (Alexa), Meta-WebIndexer and Meta-ExternalAgent (Meta AI) | The bot adds your page to an AI engine's search index, so it can be cited later. Meta-ExternalAgent also collects text to train Meta's models. |
| Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, CCBot, Bytespider | The bot collects text to train future models. It does not lead to citations today. |
Blocking the first two kinds keeps you out of AI answers. Blocking training bots is a choice some sites make on purpose. It does not stop ChatGPT search, Claude or Perplexity from citing you.
How SEOmatic Checks Your robots.txt#
- When your agents plan their work, they read your live robots.txt and check the AI search bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, Amzn-SearchBot, Amzn-User, Meta-WebIndexer, Meta-ExternalFetcher and Meta-ExternalAgent.
- A bot counts as blocked when your robots.txt shuts it out of the whole site, either by name or through the rule for all bots. SEOmatic also flags a partial block when the rules shut a bot out of pages that get your Google clicks.
- If any of them is blocked, your agents propose Prepare crawler-access changes. If only training bots are blocked, they propose nothing: keeping your content out of model training is your call. Ask your agent in chat if you want to let them in.
- The check only needs your site's address, from Google Search Console or your AI Visibility scan. Applying the fix in one click needs your website connected. If SEOmatic cannot read your robots.txt, it assumes the bots are allowed and proposes nothing.
How the Fix Lands on Each Platform#
| Platform | What happens | What you do |
|---|---|---|
| WordPress, with the SEOmatic AI plugin | SEOmatic adds the lines through the plugin, then reads your live robots.txt to prove it. If the lines do not show, it puts the previous version back. | An admin clicks Apply on my site on the Slack card. The plugin is free. See connect WordPress |
| Wix | SEOmatic edits your robots.txt through Wix, keeping every line you already have, then reads it back | An admin clicks Apply on my site on the Slack card. Your Wix connection needs the Manage SEO Settings permission |
| GitHub (custom-coded sites) | SEOmatic opens a pull request on your robots.txt file. Your first pull request opens as a draft | Merge the pull request. Nothing changes until you do |
| Shopify, Webflow, Squarespace and every other platform | Your agent writes the exact lines | Add them at the end of your robots.txt yourself |
In the dashboard, the task shows the lines under Changes to make, with Copy changes to make and an I pasted it button for when you added them yourself. The Slack card shows Exactly these lines are added at the end of your robots.txt with the Apply on my site button. See Slack. On WordPress and Wix, SEOmatic opens only the bots blocked today, keeps your rules for all bots, and adds a Sitemap line when your file has none and your site has a sitemap at the usual address. It never touches the rules for Googlebot, Bingbot or all bots, and refuses the change if it would alter what Google or Bing may crawl.
The lines open the AI search bots with Allow: /: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, Amzn-SearchBot, Amzn-User, Meta-WebIndexer, Meta-ExternalFetcher and Meta-ExternalAgent. Opening Meta-ExternalAgent also lets Meta train on your pages, because Meta uses that one bot for both. Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, CCBot and Bytespider) are left exactly as your file has them, unless you ask to allow or block them. Google-Extended only controls Gemini training and grounding; it does not affect Google Search or AI Overviews. Applebot-Extended works the same way for Apple: blocking it keeps you out of Apple's model training, not out of Siri or Spotlight.
Who can apply it
Any member can approve the task, but only a workspace Admin or Owner can click Apply on my site. It never goes live on its own, in any mode. Today the one-click button is on the Slack card; the dashboard gives you the lines to add.
When SEOmatic Will Not Apply It#
| The task says | What to do |
|---|---|
| The WordPress user is not an administrator | Reconnect with an administrator's Application Password, or add the lines by hand |
| Your site has a real robots.txt file on the server | Edit that file, or remove it so WordPress serves robots.txt, and add the lines |
| Your WordPress is set to discourage search engines | Untick it in Settings > Reading in WordPress if your site should be public |
| Your robots.txt shuts every crawler out of the whole site | That looks deliberate, so SEOmatic will not override it. Add the lines yourself if you want AI bots in |
| Your robots.txt changed since you approved this | Review the new lines and apply again |
| Your Wix connection needs the Manage SEO Settings permission | Reinstall the SEOmatic app from Connections and approve the new permission |
If you asked your agent to block AI bots instead, it writes the lines once the quality reviewer has passed them, but SEOmatic never applies a block for you. You add it yourself.
Undo#
On WordPress and Wix, the task's Undo puts your robots.txt back as it was. On GitHub, it opens a pull request that restores the file. Members, admins and the owner can undo. The verdict comes 21 days after the change is applied.
See Which Bots Actually Visit#
The AI crawlers on your site card on AI Visibility > Overview counts real visits from AI bots over the last 30 days. It does not read your robots.txt. To fill it, connect one of these:
- As an Owner or Admin, open AI Visibility. The card shows 2-minute setup with your private endpoint. Treat it like a password.
- WordPress: install the free SEOmatic AI plugin (see the SEOmatic AI WordPress plugin) and press Connect on its Home screen. AI-crawler visits then reach your dashboard on their own. You can also paste the endpoint under SEOmatic > Audit settings. It is the same plugin that applies the robots.txt fix.
- Cloudflare: click Copy Worker code (your endpoint included). In Cloudflare, go to Workers & Pages, create a Worker, paste the code and add a route for yourdomain.com/*.
Once connected, the card shows visits per kind of bot, then each bot (up to 12) with Visits, vs prior (the 30 days before), Last seen and Top page. If it says no AI-bot visits, check that your robots.txt and your CDN's bot settings do not block the crawlers you want.
- My robots.txt allows everything. Do I still need this?
- No robots.txt change is needed. If the engines still miss you, check Bing next: ChatGPT finds pages through Bing. Any member can connect IndexNow in Settings > Integrations so Bing hears about new and changed pages. See IndexNow and IndexMeNow.
- Does my CDN or firewall matter?
- Yes. Some CDN and firewall settings, like Cloudflare's AI bot blocking, stop AI bots before robots.txt is read. Once a week SEOmatic visits your homepage and one other page as ChatGPT's, Perplexity's and Claude's search bots and as a normal browser. If the bots get an error or a challenge page while the browser gets your page, your AI Visibility answers say so. The check runs from SEOmatic's servers, so a pass is not proof the real bots get in. SEOmatic does not change CDN settings; change them in your CDN.
Was This Article Helpful?
Still need help?Email contact@seomatic.ai with your site address and a screenshot. A person replies, usually within one business day.