Where Is robots.txt in WordPress?
Open yoursite.com/robots.txt in your browser. That is the file Google reads.
In most cases there is no robots.txt file on your server at all. WordPress builds a virtual file on the fly each time someone asks for it. That is why you cannot find it in your files over FTP. If someone uploaded a real robots.txt file to the root folder of your site, that physical file wins and WordPress's virtual one is ignored.
This matters when you edit it:
- Yoast SEO creates a physical file when you click "Create robots.txt file".
- Rank Math edits the virtual file, so its editor does not work while a physical file exists.
What the Default WordPress robots.txt Says
A fresh WordPress site serves this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/wp-sitemap.xmlDisallow: /wp-admin/keeps crawlers out of the dashboard.Allow: /wp-admin/admin-ajax.phplets themes and plugins that load content through admin-ajax still work for Googlebot.- The
Sitemap:line points to WordPress's core sitemap. When Yoast or Rank Math is active, the line points to theirsitemap_index.xmlinstead.
A Good robots.txt for Most WordPress Sites (Copy This)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xmlThat is it. Change the sitemap URL to the one your site really uses (open it in a browser to be sure it loads). Most sites need nothing more.
Add a rule only when you have a crawl problem you can name. Two common ones:
| Problem | Rule | When to use it |
|---|---|---|
| WooCommerce cart and checkout crawled | Disallow: /cart/ and Disallow: /checkout/ and Disallow: /my-account/ | These pages are useless in search. They also carry noindex in WooCommerce, so the rule only saves crawling |
| Endless filter URLs | Disallow: /*?filter_ (match your filter plugin's parameter) | Only if the filter URLs do not need to pass a canonical. Check first |
Rules That Hurt WordPress Sites
These come from old guides and still show up on live sites:
Disallow: /wp-content/orDisallow: /wp-includes/. These folders hold your CSS, JavaScript and images. Google needs them to render your pages. Blocking them can make Google see a broken page.Disallow: /left over from staging. One line blocks the whole site.- Blocking pages you want gone from Google. Google says robots.txt "is not a mechanism for keeping a web page out of Google" (Google Search Central). A blocked URL can still show in results if other pages link to it. Worse, Google cannot see a noindex tag on a page it is not allowed to crawl. To remove a page, use noindex and let Google crawl it.
- Blocking
/tag/or/category/. Same problem. If you do not want archive pages in Google, set them to noindex in your SEO plugin instead. Noindex:lines. Google stopped supporting noindex inside robots.txt in 2019. The line does nothing.
"Discourage Search Engines" Does Not Change robots.txt Anymore
Since WordPress 5.3, ticking Settings > Reading > "Discourage search engines from indexing this site" adds a noindex, nofollow robots meta tag to every page. It no longer writes Disallow: / into robots.txt (WP Tavern). So if your site is missing from Google and robots.txt looks fine, check that box next.
How to Edit robots.txt in WordPress
| Method | Path | Notes |
|---|---|---|
| Yoast SEO | Yoast SEO > Tools > File editor | Click "Create robots.txt file" if none exists. Needs file editing to be allowed on your host |
| Rank Math | Rank Math SEO > General Settings > Edit robots.txt | Needs Advanced Mode. Does not work if a physical robots.txt file exists: delete that file first |
| All in One SEO | All in One SEO > Tools > Robots.txt Editor | Builds rules from a form |
| FTP or host file manager | Upload robots.txt to the folder that holds wp-config.php | Always works. The physical file overrides every plugin |
After any edit, open yoursite.com/robots.txt in a private window to see what is live. If you use a page cache or CDN, purge it.
Should You Block AI Crawlers in WordPress robots.txt?
It depends on what you want, and each company now splits its crawlers by job:
| Bot | Company | What it does | Block it if |
|---|---|---|---|
| GPTBot | OpenAI | Collects data for model training | You do not want your text used for training |
| OAI-SearchBot | OpenAI | Powers ChatGPT search results and citations | Almost never: blocking it can keep you out of ChatGPT answers |
| ClaudeBot | Anthropic | Training | You do not want your text used for training |
| Claude-SearchBot | Anthropic | Search indexing for Claude answers | Almost never |
| Google-Extended | Controls use in Gemini models. Does not affect Google Search | You want out of Gemini training, still in Search |
Sources: OpenAI bots, Google's common crawlers, Search Engine Journal on Anthropic's three bots.
A common middle ground: block training bots, allow search bots.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Also check your security plugin, host firewall or CDN. Some block AI bots by default, which overrides whatever robots.txt says.
How to Test Your robots.txt
- Open Search Console > Settings > robots.txt. It shows the file Google last fetched, when, and any errors.
- Use URL Inspection on one key page. If it says "Blocked by robots.txt", find the rule that matches.
- Check Search Console > Indexing > Pages for "Blocked by robots.txt". Every URL on that list should be one you meant to block.
How to Use This with an AI Assistant
Connect SEOmatic at https://seomatic.ai/mcp and ask: "Use check_site on mysite.com and tell me whether my robots.txt blocks Googlebot or the AI search crawlers. Show me the exact lines to change." check_site is free and needs no sign-in.
Let the Agents Do This on Your WordPress Site
SEOmatic's AI agents connect to WordPress, read your own Google Search Console data, and make the fixes from this checklist directly on your site:
- Rewrite SEO titles and meta descriptions, including through Yoast or Rank Math
- Add internal links and schema where they are missing
- Set up redirects and fix pages blocked from Google
Each change can wait for your approval, one click undoes it, and the agents check it against how the rest of your site did. What they can change depends on your platform, and you see the full list before anything runs.
14-Day Trial for $1, credited to your first payment.