The 14 areas of a technical SEO audit
| # | Area | The question it answers | Main tool |
|---|---|---|---|
| 1 | Crawlability | Can Googlebot reach the page? | robots.txt report, Crawl stats |
| 2 | Indexability | Is Google allowed to index it? | URL Inspection, view source, curl -I |
| 3 | Redirects and status codes | Does each URL return the right code in one hop? | A crawler (Screaming Frog, Sitebulb) |
| 4 | JavaScript rendering | Is the content there after rendering, and before it? | URL Inspection live test, curl |
| 5 | Site architecture | Can Google find every page through links? | Crawl depth and orphan reports |
| 6 | Canonicals | Does every signal name the same preferred URL? | URL Inspection (user vs Google canonical) |
| 7 | Hreflang | Do language versions point to each other correctly? | Crawler hreflang report |
| 8 | Structured data | Is the markup valid and true to the page? | Rich Results Test |
| 9 | Core Web Vitals | Is the page fast and stable for real users? | Core Web Vitals report, PageSpeed Insights |
| 10 | Mobile | Does the mobile version have everything? | Smartphone URL Inspection, Lighthouse |
| 11 | HTTPS and security | Is every page secure and clean? | HTTPS report, Security & Manual Actions |
| 12 | Server logs | What did crawlers really request? | Access logs, log analyzer |
| 13 | AI crawlers | Can AI search tools read and cite you? | robots.txt, WAF bot settings |
| 14 | Monitoring | Will you notice when a release breaks something? | Scheduled crawls, alerts |
Start here: the 10 checks that break sites most often
This is a real excerpt from the spreadsheet. If you only have one hour, do these.
| Check | How to check | How to fix |
|---|---|---|
| robots.txt returns 200 (or 404), never 5xx | Open /robots.txt. Search Console > Settings > robots.txt report | Fix the server rule. On a 5xx, Google stops crawling for 12 hours |
| robots.txt does not block pages, CSS or JS you need | Read every Disallow line. URL Inspection > Page resources | Remove broad rules. Block only cart, internal search, admin |
| No accidental noindex on pages that should rank | View source for "noindex". URL Inspection | Remove the tag or the CMS setting that adds it |
| No noindex in the X-Robots-Tag header | curl -I https://yoursite.com/page | Remove the server or CDN rule |
| Only one host version (https, www or non-www) | Open all 4 versions | 301 each to the chosen one in one hop |
| Removed pages return 404 or 410, not 200 | Open a made-up URL, check status | Make the error template send a real 404 |
| Canonicals point to live 200 URLs | Crawler canonical report | Point to the final live URL |
| Firewall and CDN let Googlebot through | Crawl stats > Host status. WAF logs | Allowlist verified crawlers (reverse DNS) |
| Main content is in the rendered HTML | URL Inspection > View tested page | Server-side render key templates |
| Sitemap lists only canonical 200 URLs | Crawl the sitemap URLs | Regenerate it from your CMS |
What changed in technical SEO from 2024 to 2026
Many top-ranking checklists still repeat old advice. Here is what is true in October 2026.
| Change | Date | What to do | Source |
|---|---|---|---|
| INP replaced FID as a Core Web Vital | March 12, 2024 | Measure INP (good is 200 ms or less). Ignore FID advice | web.dev |
| Mobile Usability report and Mobile-Friendly Test retired | December 1, 2023 | Use Lighthouse and real phones | |
| Breadcrumb rich results shown on desktop only | January 2025 | Keep breadcrumbs for users and links. Do not expect them on mobile results | Google updates |
| Googlebot fetches the first 2 MB of HTML for Search | Documented February 2026 | Keep key content early. Move large inline data out of the HTML | Googlebot docs |
| FAQ rich results stopped showing | May 7, 2026 | Stop adding FAQ markup for rich results. Keep useful FAQs on the page | Google updates |
| llms.txt not needed for Google Search | June 2026 | Optional. No ranking effect in Google | Google updates |
| No special markup needed for AI Overviews and AI Mode | Current | A page must be indexed and eligible for a snippet. That's it | Google AI features |
How to run a technical SEO audit in about 2 hours
- Open Search Console (a Domain property). Check Settings > Crawling > Crawl stats and the robots.txt report. Note any 5xx, host problems or a crawl drop.
- Read your robots.txt. Every Disallow line should block only junk. Check the AI crawler lines too.
- Run a crawl of up to a few thousand URLs with any crawler. Export status codes, redirects, canonicals, indexability, depth and orphan pages.
- Inspect your 10 most important URLs with URL Inspection. Compare user-declared and Google-selected canonical. Look at the rendered HTML and screenshot.
- Open the Page indexing report. Only care about reasons that hit pages you want in Google. Our Page indexing fix guide covers each reason.
- Open the Core Web Vitals report. Fix the URL groups marked Poor first, mobile before desktop.
- Test one URL per template (home, category, product or service, post) in the Rich Results Test and PageSpeed Insights.
- Mark each row in the spreadsheet as Done, To do or Not needed. The Progress sheet shows what is left.
Worked example (fictional site). northwind-outdoors.com moved to a new theme. Clicks fell 30% in two weeks. The audit found two problems in 20 minutes: the new theme added a second canonical tag that pointed every product to the homepage, and the CDN's new bot rule was challenging Googlebot (Crawl stats showed a jump in 5xx). Removing the duplicate canonical and allowlisting verified Googlebot IPs fixed both. Neither would have shown up in an "on-page" checklist.
Technical SEO for AI search
AI answers are built from pages that crawlers can fetch. Two things matter.
1. Let the right crawlers in. Search crawlers decide if you can be cited. Training crawlers are a separate choice.
| User agent | Company | Purpose | Our default |
|---|---|---|---|
| Googlebot | Search, AI Overviews, AI Mode | Allow | |
| Google-Extended | Gemini training and grounding. No effect on Google Search | Your choice | |
| OAI-SearchBot | OpenAI | ChatGPT search answers | Allow |
| GPTBot | OpenAI | Model training | Your choice |
| Claude-SearchBot | Anthropic | Claude search results | Allow |
| ClaudeBot | Anthropic | Model training | Your choice |
| PerplexityBot | Perplexity | Perplexity search results | Allow |
OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers (OpenAI docs). Check your firewall too: some CDN "block AI bots" settings catch search crawlers along with training ones.
2. Serve the main content as HTML. Many AI crawlers fetch the raw HTML and do not run JavaScript. If curl https://yoursite.com/page does not show your main text, they may see an empty page.
What this checklist does not cover
This is about how your site works. Other SEOmatic resources go deeper on the rest:
- Each Page indexing reason (crawled or discovered not indexed, soft 404, duplicates): Page indexing fix guide
- Search Console settings and reports, row by row: Search Console audit checklist
- Internal links and orphan pages in depth: Internal linking playbook
- Platform-specific settings: WordPress and Shopify SEO checklists
- Titles and meta descriptions: Title and meta swipe file
Who this is for
- Site owners and marketers who want a clear list they can hand to a developer, with the "where to check" spelled out.
- Freelancers and agencies who need a repeatable technical audit template for client sites.
- Developers who want the Google rules (status codes, canonicals, rendering, hreflang) in one place with sources.
How to use this with an AI assistant
You can run this checklist with ChatGPT, Claude, Cursor or any MCP client connected to SEOmatic. Connect SEOmatic at https://seomatic.ai/mcp, then paste one of the prompts below.
The free check needs no account. Reading your Search Console data is free after sign-in. Staging fixes needs the 14-day trial, and nothing goes live until you approve it.
- No sign-in: check_site gives a free first look at one site.
- After free sign-in: your assistant can read your Search Console data and audit your pages.
- Staging fixes needs the 14-day trial. Nothing goes live until you approve it, and every applied change can be undone.
What the tools cannot do (so your assistant does not pretend): they do not read server logs, run Lighthouse or field Core Web Vitals data, or test hreflang sets across a whole site. site_audit covers up to 30 pages per run. For those rows of the checklist, the assistant should tell you which tool to use (listed in the spreadsheet) instead of guessing.
Copy-paste prompt
Use SEOmatic to run the technical SEO checklist on mysite.com.
1. Run check_site on mysite.com. Report the homepage status, redirects, noindex, canonical, robots.txt rules for Googlebot and AI crawlers, JavaScript-only rendering, and the sitemap sample.
2. Sign me in with connect_seomatic, then call get_started for the overview.
3. Run site_audit. Group what it finds into the checklist sections: Indexability (noindex leaks), Canonicals (canonicals pointing elsewhere), Redirects and status codes (dead pages, broken internal links), Content (missing or duplicate titles and metas, near-empty pages), Speed (slow pages).
4. Check sitemaps with gsc_insights (action: sitemaps).
5. Take my 20 most important URLs (from gsc_performance, action: top_pages, last 28 days) and inspect them with gsc_indexing (action: batch_inspect). For each, compare the user-declared and Google-selected canonical and flag any page that is not indexed.
6. For any template that looks wrong, run site_pages (action: analyze) on one example URL and site_pages (action: crawl) to show its title, canonical, headings, links and schema.
7. Check for duplicate pages competing for the same searches with gsc_insights (action: cannibalization).
8. If AI crawler tracking is set up, call get_ai_crawler_activity and tell me which AI bots visit and whether robots.txt lets them in.
9. Give me one table: section, check, evidence (with the URL), fix, priority (High/Medium/Low). High = stops crawling or indexing, or affects every page. Say plainly which checklist rows you could not check with these tools.
10. For fixes SEOmatic can make, stage them with task_manage (action: create), up to 5 per call, then call show_approval_cards so I can approve or dismiss each one. Do not change anything live without my approval. If I want an applied change back, use task_manage (action: rollback).Tools this prompt uses
| MCP tool (action) | What it does |
|---|---|
| check_site | Free, no sign-in: homepage status, redirects, noindex, canonical, robots.txt for Googlebot and AI crawlers, JavaScript-only rendering, up to 10 sitemap pages, top fixes |
| connect_seomatic | Shows the free sign-in prompt |
| get_started | Overview of your site: verdict and top 3 actions with numbers |
| site_audit | Crawls up to 30 pages: duplicate titles and metas, missing metas, near-empty pages, noindex leaks (meta or X-Robots-Tag), canonicals pointing elsewhere, slow pages, dead pages, broken internal links |
| gsc_insights (sitemaps) | Health of every submitted sitemap: errors, warnings, pending, stale |
| gsc_insights (cannibalization) | Queries where two of your pages compete |
| gsc_performance (top_pages) | Your top pages by clicks from Search Console |
| gsc_indexing (batch_inspect) | Google's index status for up to 50 URLs per call. Google caps URL Inspection at 2,000 requests per day per property |
| site_pages (analyze) | One page: 0-100 score and issues ranked by impact |
| site_pages (crawl) | One page: title, meta, headings, links, images, schema, Open Graph |
| get_ai_crawler_activity | Which AI bots visit your site and how. Needs a one-time setup (WordPress plugin or Cloudflare Worker); otherwise it says visits are unmeasured and checks robots.txt access |
| task_manage (create, decide, rollback) | Stages fixes as proposals (never runs them), records your approve or dismiss, undoes an applied change |
| show_approval_cards | Shows Approve and Dismiss cards for staged fixes in the chat |
Which checklist fixes SEOmatic can stage
Staged as tasks for your approval (applied in your CMS where the connection allows it, otherwise you get the exact change to paste): noindex removal, canonical fixes, broken internal link fixes, redirects (on Shopify, BigCommerce, Wix, HubSpot, Framer, Drupal, and WordPress with the SEOmatic Redirects plugin), robots.txt rules, structured data (schema and repairs), hreflang, image dimensions and LCP loading fixes (cwv_fix), alt text, headings, sitemap submission and index requests in Search Console.
Not done by SEOmatic: server, CDN or firewall settings, hosting speed, HTTPS certificates, JavaScript framework changes (SSR), and server log analysis.
Go deeper
- Guiderobots.txt for AI Crawlers: Which to Allow and Which to Block
- GuideHow to Fix INP (Interaction to Next Paint)
- GuideHow to Do a Technical SEO Audit, Step by Step (2026)
- GuideJavaScript SEO Checklist for React, Next.js and Single-Page Apps
- GuideHreflang Checklist: Set It Up Right and Fix Common Errors
- GuideStructured Data Checklist 2026: What Still Earns a Rich Result
Sources (24)
- developers.google.com/search/docs/crawling-indexing/robots/robots_txt
- developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
- developers.google.com/search/docs/crawling-indexing/googlebot
- developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- developers.google.com/search/docs/crawling-indexing/verifying-googlebot
- developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
- developers.google.com/search/docs/crawling-indexing/301-redirects
- developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
- developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics
- developers.google.com/search/docs/specialty/international/localized-versions
- developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing
- developers.google.com/search/docs/appearance/page-experience
- web.dev/articles/vitals
- web.dev/blog/inp-cwv-march-12
- developers.google.com/search/blog/2023/04/page-experience-in-search
- developers.google.com/search/blog/2023/10/mobile-first-is-here
- developers.google.com/search/updates
- developers.google.com/search/blog/2023/08/howto-faq-changes
- developers.google.com/search/docs/appearance/structured-data/sd-policies
- developers.google.com/search/docs/appearance/ai-features
- developers.google.com/search/docs/fundamentals/ai-optimization-guide
- developers.openai.com/api/docs/bots
- support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- docs.perplexity.ai/guides/bots


