Robots.txt Tester
Test whether Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot or another crawler is allowed to fetch specific paths. See the exact winning rule, group selection, sitemap directives and parser warnings.
RFC 9309 matchingGoogle-style group precedenceBulk pathsNo signup
Fetching and parsing robots.txt…
Crawler access report
robots.txt test
Path results
Allowed or blocked?
Each result shows the winning rule that determined crawler access.
Parser evidence
Matched crawler group
Validation
Syntax and directive warnings
Unsupported directives are reported without pretending they are crawler rules.
Discovery
Sitemap directives
Absolute sitemap URLs declared in the file.
Source robots.txt
Methodology and limitations
How robots rules work
Test robots.txt rules before they block important pages
A correct robots.txt file controls crawling by compliant bots; it does not protect private content and does not reliably remove URLs from search results.
How this robots.txt tester chooses the winning rule
The evaluator first selects the most specific matching user-agent group. If multiple groups target the same most-specific crawler token, their rules are combined. A specific crawler group is not merged with the global User-agent: * group. Among matching Allow and Disallow paths, the most specific path wins; when equally specific rules conflict, Allow wins.
Googlebot
Use this mode to test ordinary Google Search crawling against Googlebot-specific or global rules.
Bingbot
Check whether Bing's crawler matches a dedicated Bingbot group or falls back to the global rules.
AI crawler testing
Test GPTBot, ClaudeBot, PerplexityBot and Google-Extended against the rules you publish. A crawler's actual compliance remains the crawler operator's responsibility.
Bulk URL paths
Paste multiple paths or full URLs to identify broad rules that unintentionally block sections of a site.
Wildcard matching
* matches zero or more characters and terminal $ anchors the end of the path. More specific matching paths outrank shorter matches.
Exact rule evidence
Instead of only returning “blocked,” the tool shows the winning directive, source line and all matching rules considered.
Robots.txt does not mean noindex
A disallowed URL may still be known from links and can potentially appear in search results without its content being crawled. If the goal is to prevent indexing, use an appropriate indexing directive on a crawlable response and verify it with the search engine's inspection tools. Do not use robots.txt as authentication or access control.
What about crawl-delay?
This tester flags Crawl-delay as unsupported by the Google-style rule evaluator because Google documents user-agent, allow, disallow and sitemap as its supported robots fields. Other crawlers may implement additional directives differently.
Related tools
Primary references
RFC 9309 — Robots Exclusion Protocol · Google robots.txt specification