Free Robots.txt Validator

Our robots.txt validator parses your robots.txt the way a search crawler does and tells you exactly what it permits. Paste the file and the tool groups the rules by user-agent, lists every Allow and Disallow, surfaces the Sitemap and Crawl-delay directives, and flags syntax errors and warnings. Then you can test any URL against the rules and see whether it is allowed or blocked, and which line made the call. It is built for SEOs and developers who need to confirm a page is crawlable before, or after, a deploy.

The matching follows Google's documented behavior: the most specific (longest) matching rule wins, and on a tie an Allow beats a Disallow. Wildcards (*) and end-of-path anchors ($) are supported, and URL paths are tested with their query string, so a rule like "Disallow: /*.pdf$" blocks a .pdf page but not a .pdf URL with parameters. Everything runs locally in your browser, so nothing about your site is sent to a server.

For SEOs

Our tool turns a guessing game into a clear answer, so you can:

• Confirm an important page is not accidentally blocked
• Test a specific URL against the live rules for Googlebot or any crawler
• Catch a stray Disallow before it deindexes a section

For Developers

Our tool gives you a fast pre-deploy check, so you can:

• Validate robots.txt syntax and spot ignored or misplaced directives
• See exactly which rule applies to a path
• Verify Allow overrides and wildcard rules behave as intended

Why Use Our Robots.txt Validator

• Real crawler logic: The URL tester uses Google's longest-match and Allow-over-Disallow rules with wildcard and anchor support, so the verdict reflects how search engines actually read your file, not a naive prefix check.
• Catches the silent mistakes: Rules before a User-agent line, unknown directives, relative sitemap URLs, and malformed paths are all flagged, the kind of errors that quietly stop a crawler without any visible sign.
• Private by default: Parsing and testing happen in your browser. Your robots.txt is never uploaded to a server.