Where to find the report
- In Search Console, click Settings in the left menu.
- Under Crawling, find Crawl stats and click Open report.
The report covers the last 90 days. It only exists for root-level properties: a Domain property (example.com) or a root URL-prefix property (https://example.com). A property for a folder, like https://example.com/blog/, does not have it (Crawl Stats report help).
Most site owners never open it, because it sits under Settings. That is a mistake. It is where server and firewall problems show up first, often days before clicks drop.
Check 1: Host status
The Host status panel tells you if Google could reach your site. Google uses three icons:
| Icon | What Google says | What to do |
|---|---|---|
| Green | No significant crawl availability issues in the past 90 days | Nothing |
| Yellow | At least one significant issue in the past 90 days, but more than a week ago | Click in and see what happened and when |
| Red | At least one significant issue in the last week | Fix now |
Click the panel to see the three things Google checks:
- robots.txt fetching. Every time Google crawls, it first fetches your robots.txt. A 200 with a valid file is fine. A 404 (no file) is also fine. A server error or timeout is not. If robots.txt fails, Google stops crawling for the first 12 hours. From 12 to 30 days it uses the last good copy. After 30 days it acts as if there is no robots.txt, if your homepage loads (Google).
- DNS resolution. Your DNS server did not recognize your hostname or did not answer. Common after a domain or DNS provider change.
- Server connectivity. Your server did not respond, or did not send a full response. Common on overloaded shared hosting.
Each chart has a red dotted line. Days above it count as a problem.
Check 2: Crawl requests by response
The By response table shows what your server sent back. Google counts these as good:
- OK (200)
- Moved permanently (301) and moved temporarily (302)
- Not modified (304)
Watch these:
- Server error (5xx): your server failed. Look at the dates and match them with deploys or traffic spikes.
- Unauthorized (401/407) and other 4xx: often a login wall or a bot rule.
- Page timeout, fetch error, page could not be reached: your server or a proxy was too slow or dropped the connection.
- robots.txt not available: see Check 1.
- Not found (404): fine for pages you deleted on purpose. A sudden jump after a redesign means links point to URLs that no longer exist.
Click any row to see example URLs.
Check 3: Average response time
The main chart shows total crawl requests, total download size and average response time. Response time here includes every file Google fetched, not just HTML pages: each image, script and stylesheet counts (Google).
Google does not publish a target number. Watch the trend. If response time doubles and crawl requests fall in the same week, your server is struggling. Google crawls less when a server is slow, so new and updated pages get found later.
Check 4: File type, purpose and Googlebot type
Three more tables:
| Table | What it shows | What to look for |
|---|---|---|
| By file type | HTML, images, JavaScript, CSS, other | A sudden rise in "Other" or JSON can mean Google is crawling API or filter URLs |
| By purpose | Discovery (URLs Google never crawled before) and Refresh (recrawls of known pages) | A new section with almost no Discovery requests may not be linked well |
| By Googlebot type | Smartphone, Desktop, Image, Video, AdsBot and others | Most requests should come from the smartphone crawler |
Spotting a firewall or CDN that blocks Googlebot
This is the most common serious problem the report reveals. Security tools, bot-protection rules and some hosts block or challenge Googlebot by mistake. The site works for you, but not for Google.
Signs, usually together:
- Host status turns yellow or red.
- A spike in 4xx (often 403), 5xx or timeouts in By response.
- Total crawl requests drop sharply.
- Days later, pages move to "Server error (5xx)" or "Blocked due to access forbidden (403)" in Indexing > Pages.
What to do:
- Run URL Inspection > Test live URL on your homepage. If it fails while the page loads fine in your browser, something is blocking Google.
- Check your firewall, CDN or security plugin logs for blocked requests from Googlebot. Verify they are really Google using Google's verifying Googlebot guide, then allow them.
- Ask your host whether they rate-limit crawlers.
Worked example (fictional site). northwind-outdoors.com turned on a new bot-protection setting at its CDN on a Friday. The next week, Host status went red under Server connectivity, and By response showed a wall of "Page could not be reached". Crawl requests fell from about 4,000 a day to under 300. URL Inspection's live test failed on the homepage. The team added an allow rule for verified Googlebot traffic, the live test passed the same day, and crawl requests climbed back over the next week.
What Crawl Stats does not tell you
- It does not say which pages are indexed. Use Indexing > Pages for that, and our indexing problems guide for each reason.
- It does not show crawl from other search engines or AI crawlers.
- More crawling is not better by itself. A small site with a stable crawl and green host status is healthy.
If you see too much crawling that hurts your server, Google suggests returning 503 or 429 for a short time, not blocking Googlebot for good.
How to use this with an AI assistant
Connect SEOmatic at https://seomatic.ai/mcp and ask: "Run check_site on mysite.com and tell me if Googlebot can fetch my homepage and robots.txt. Then use gsc_indexing inspect on my homepage and my top product page and report the crawl status and last crawl date." check_site is free and needs no sign-in; SEOmatic does not read the Crawl Stats report itself, so open it under Settings for the 90-day view.