WordPress robots.txt: What It Should Say and How to Edit It

Where is robots.txt in WordPress?

Open yoursite.com/robots.txt in your browser. That is the file Google reads.

In most cases there is no robots.txt file on your server at all. WordPress builds a virtual file on the fly each time someone asks for it. That is why you cannot find it in your files over FTP. If someone uploaded a real robots.txt file to the root folder of your site, that physical file wins and WordPress's virtual one is ignored.

This matters when you edit it:

• Yoast SEO creates a physical file when you click "Create robots.txt file".
• Rank Math edits the virtual file, so its editor does not work while a physical file exists.

What the default WordPress robots.txt says

A fresh WordPress site serves this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/wp-sitemap.xml

• Disallow: /wp-admin/ keeps crawlers out of the dashboard.
• Allow: /wp-admin/admin-ajax.php lets themes and plugins that load content through admin-ajax still work for Googlebot.
• The Sitemap: line points to WordPress's core sitemap. When Yoast or Rank Math is active, the line points to their sitemap_index.xml instead.

A good robots.txt for most WordPress sites (copy this)

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap_index.xml

That is it. Change the sitemap URL to the one your site really uses (open it in a browser to be sure it loads). Most sites need nothing more.

Add a rule only when you have a crawl problem you can name. Two common ones:

Rules that hurt WordPress sites

These come from old guides and still show up on live sites:

1. Disallow: /wp-content/ or Disallow: /wp-includes/. These folders hold your CSS, JavaScript and images. Google needs them to render your pages. Blocking them can make Google see a broken page.
2. Disallow: / left over from staging. One line blocks the whole site.
3. Blocking pages you want gone from Google. Google says robots.txt "is not a mechanism for keeping a web page out of Google" (Google Search Central). A blocked URL can still show in results if other pages link to it. Worse, Google cannot see a noindex tag on a page it is not allowed to crawl. To remove a page, use noindex and let Google crawl it.
4. Blocking /tag/ or /category/. Same problem. If you do not want archive pages in Google, set them to noindex in your SEO plugin instead.
5. Noindex: lines. Google stopped supporting noindex inside robots.txt in 2019. The line does nothing.

"Discourage search engines" does not change robots.txt anymore

Since WordPress 5.3, ticking Settings > Reading > "Discourage search engines from indexing this site" adds a noindex, nofollow robots meta tag to every page. It no longer writes Disallow: / into robots.txt (WP Tavern). So if your site is missing from Google and robots.txt looks fine, check that box next.

How to edit robots.txt in WordPress

After any edit, open yoursite.com/robots.txt in a private window to see what is live. If you use a page cache or CDN, purge it.

Should you block AI crawlers in WordPress robots.txt?

It depends on what you want, and each company now splits its crawlers by job:

Sources: OpenAI bots, Google's common crawlers, Search Engine Journal on Anthropic's three bots.

A common middle ground: block training bots, allow search bots.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Also check your security plugin, host firewall or CDN. Some block AI bots by default, which overrides whatever robots.txt says.

How to test your robots.txt

1. Open Search Console > Settings > robots.txt. It shows the file Google last fetched, when, and any errors.
2. Use URL Inspection on one key page. If it says "Blocked by robots.txt", find the rule that matches.
3. Check Search Console > Indexing > Pages for "Blocked by robots.txt". Every URL on that list should be one you meant to block.