Google's own help notes that URLs blocked by robots.txt are often blocked on purpose, so not every one needs a fix (Google, Page indexing report help). This is one of 17 reasons in the Page indexing report. See every reason and whether to fix it.
What the Status Means
robots.txt is a plain text file at yourdomain.com/robots.txt. It tells crawlers which paths they may not visit:
User-agent: *
Disallow: /cart
Disallow: /searchAny URL that matches a Disallow line for Googlebot lands in this row. Google says robots.txt is used mainly to avoid overloading your site with requests, and that "it is not a mechanism for keeping a web page out of Google" (Google, robots.txt introduction).
Fine or a Problem?
| Blocked URL | Normal? | Action |
|---|---|---|
| Cart, checkout, account, admin | Yes | None |
| Internal search results, filter URLs | Yes | None |
| A product, service, blog or location page | No | Remove the block |
Every page on the site (Disallow: /) | No, unless the site is not live yet | Remove the block at launch |
| CSS or JavaScript files | No | Allow them, so Google can render your pages |
How to Find the Rule That Blocks a Page
- Open
yourdomain.com/robots.txtand read theDisallowlines underUser-agent: *andUser-agent: Googlebot. - Match the page path against each rule.
Disallow: /blogalso blocks/blog-tipsand/blog/anything, because rules match the start of the path. - In Search Console, open Settings > robots.txt to see the file Google last fetched and any errors it found.
- Run URL Inspection on the page. The crawl section shows if robots.txt blocked it.
How to Fix It, by Platform
| Platform | Where robots.txt is edited |
|---|---|
| WordPress | Yoast: Tools > File editor. Rank Math: General Settings > Edit robots.txt. See WordPress robots.txt |
| Shopify | The robots.txt.liquid theme template. See Shopify robots.txt |
| Wix | The robots.txt editor in the SEO settings. See Wix robots.txt |
| Webflow | Site settings > SEO > robots.txt |
| Squarespace | Not editable. Squarespace sets the file for you |
After the fix, run URL Inspection on the page and click Request indexing, then click Validate fix on the row in the Page indexing report.
Do Not Use robots.txt to Hide a Page
A blocked URL can still be indexed if other pages link to it, without its content. That is the Indexed, though blocked by robots.txt warning. To keep a page out of Google, allow crawling and use a noindex tag, or put the page behind a password.
How to Use This with an AI Assistant
Connect SEOmatic at https://seomatic.ai/mcp and ask: "Sign me in with connect_seomatic. Use gsc_indexing (action: inspect) on https://mysite.com/page and tell me if robots.txt blocks it, and which rule. Then draft the corrected robots.txt for my approval." In Approval mode, nothing on your site changes until you approve.
Let the Agents Fix Your Indexing Problems
SEOmatic's AI agents read your own Google Search Console data and fix what keeps pages out of Google, directly in your CMS:
- Remove stray noindex tags and correct canonicals on pages that should rank
- Fix robots.txt rules that block pages you want in Google
- Submit your sitemap and request indexing for pages that are ready
Each change can wait for your approval, one click undoes it, and the agents check it against how the rest of your site did. What they can change depends on your platform, and you see the full list before anything runs.
14-Day Trial for $1, credited to your first payment.
More From the Page Indexing Fix Guide
- The GuidePage Indexing: Why Pages Aren't Indexed
- GuideCrawled, Currently Not Indexed: The Fix
- GuideDiscovered, Currently Not Indexed: The Fix
- GuideSoft 404: What It Is and How to Fix It
- GuidePage Is Not Indexed: Page with Redirect
- GuideAlternate Page with Proper Canonical Tag: Meaning
- GuideDuplicate Without User-Selected Canonical: The Fix
- GuideWhy Is My Website Not Showing Up on Google?
- GuideExcluded by Noindex Tag: What It Means and the Fix
- GuideIndexed, Though Blocked by robots.txt
- The ChecklistSite Migration SEO Checklist