Google explains it plainly: "we might still find and index a disallowed URL if it is linked from other places on the web" (Google, robots.txt introduction). This status shows as a warning on the indexed side of the Page indexing report. For every other reason, see the Page indexing fix guide.
Why Google Indexes a Page It Cannot Crawl
robots.txt controls crawling, not indexing. When Google finds links to a blocked URL, it can still add the address to its index, using the link text as a hint. It cannot read the page, so it cannot see a noindex tag either. The search result often says no information is available for the page.
Pick One of Two Fixes
| You want the page | Do this |
|---|---|
| On Google | Remove the robots.txt rule that blocks it. Then request indexing in URL Inspection |
| Off Google | Add a noindex tag, then remove the robots.txt block so Google can crawl the page and see the tag. Once it drops out, you may block it again |
| Private (an admin or staging page) | Put it behind a password. A password beats both |
The common mistake is to add a noindex and keep the block. Google cannot see the tag through the block, so nothing changes. Google says noindex works only when the page "must not be blocked by a robots.txt file" (Google, block indexing with noindex).
Common Causes, by Platform
- Shopify: the default robots.txt blocks cart, checkout and filtered collection URLs, and links to them can still get indexed. Usually harmless. See Shopify robots.txt.
- WordPress: a
Disallowrule added by a plugin or by hand that also catches real pages, likeDisallow: /?or a folder. See WordPress robots.txt. - Any site: a staging or old folder blocked in robots.txt but still linked from the live site.
How to Check After the Fix
- Run URL Inspection on a sample URL and click Test live URL.
- Confirm crawling is allowed and, if you added one, that the noindex is seen.
- Click Validate fix on the row in the Page indexing report.
The related status Blocked by robots.txt is the same block on a URL Google did not index.
How to Use This with an AI Assistant
Connect SEOmatic at https://seomatic.ai/mcp and ask: "Sign me in with connect_seomatic. Use gsc_indexing (action: batch_inspect) on these URLs, tell me which are indexed though blocked by robots.txt, and whether each should be on Google. Then draft the robots.txt and noindex changes for my approval." In Approval mode, nothing on your site changes until you approve.
Let the Agents Fix Your Indexing Problems
SEOmatic's AI agents read your own Google Search Console data and fix what keeps pages out of Google, directly in your CMS:
- Remove stray noindex tags and correct canonicals on pages that should rank
- Fix robots.txt rules that block pages you want in Google
- Submit your sitemap and request indexing for pages that are ready
Each change can wait for your approval, one click undoes it, and the agents check it against how the rest of your site did. What they can change depends on your platform, and you see the full list before anything runs.
14-Day Trial for $1, credited to your first payment.
More From the Page Indexing Fix Guide
- The GuidePage Indexing: Why Pages Aren't Indexed
- GuideCrawled, Currently Not Indexed: The Fix
- GuideDiscovered, Currently Not Indexed: The Fix
- GuideSoft 404: What It Is and How to Fix It
- GuidePage Is Not Indexed: Page with Redirect
- GuideAlternate Page with Proper Canonical Tag: Meaning
- GuideDuplicate Without User-Selected Canonical: The Fix
- GuideWhy Is My Website Not Showing Up on Google?
- GuideBlocked by robots.txt: What It Means and the Fix
- GuideExcluded by Noindex Tag: What It Means and the Fix
- The ChecklistSite Migration SEO Checklist