The AI Bot Robots.txt Checker applies the standardized matching logic of RFC 9309 to one pasted robots.txt file and reports an allowed or blocked decision for 32 documented AI crawler, assistant, training-control, and AI-adjacent search product tokens against a single URL path you enter. The tool runs entirely in your browser, never fetches your live website, never uploads your file, and never contacts a crawler operator, which means the result is fully reproducible from the same text and path but cannot detect deployment, redirect, caching, or host-scope problems on your real server. When a check looks wrong, the problem almost always falls into one of four buckets: a misunderstanding of how the parser handles specific versus wildcard groups, a case-sensitive path comparison, a rule set that was rejected by a safety limit rather than the byte limit, or the assumption that a voluntary protocol decision is the same as enforced access control. Treating each bucket as a separate diagnostic step turns a confusing report into an actionable pre-deployment review.
What the Checker Reports and What It Cannot Tell You
The checker accepts at most 500 KiB of UTF-8 input and 5,000 non-empty access rules, and it runs the same parsing rules on every submission so two identical inputs always produce identical outputs. It evaluates only the rules you paste, so a discrepancy between the report and the behavior of a live crawler usually means the deployed file differs from the file under review. Because the tool does not fetch URLs, it cannot detect redirects, CDN overrides, syntax served only to certain clients, incorrect host scope, caching delays, or an unreachable file. It also cannot confirm what is currently deployed on your origin server. Those limits keep the tool private and deterministic while making the result easy to reproduce from the same text and path, and they define the boundary between what the checker can troubleshoot and what must be verified by other means.
The 32 rows in the report are robots.txt product tokens, not a promise that every string appears verbatim as an HTTP User-Agent header. Some operators publish a dedicated control token, such as Google-Extended or Applebot-Extended, for model-training or data-use preferences, while other rows represent crawler or assistant retrieval products. Each row is labeled with operator, purpose category, and source status so you can tell a first-party verified token apart from a supplemental directory entry at a glance.
Common Problems When Checking AI Bot Robots.txt
The table below maps the most frequent issues to their underlying cause and the first thing to try. It is built from the parser rules documented in RFC 9309 and from the documented limits of the checker itself.
| Problem | Likely cause | First thing to try |
|---|---|---|
| Result is allowed for a path you expected to be blocked | A matching product-token group has an Allow rule that overrides Disallow, or the wildcard group falls back when no specific group matches | Inspect both the product-token group and the wildcard group in the pasted file |
| Different answer for /Private versus /private | Path matching is case-sensitive in RFC 9309 | Re-enter the path in the exact case the crawler would request |
| File rejected even though it is under 500 KiB | The aggregate matching-work safety limit was hit by a long path combined with a large rule set | Trim the rule set or shorten the test path, then resubmit |
| Report says blocked but the crawler still visits | Robots.txt is a voluntary crawler request, not an access-control rule | Check server logs for the actual User-Agent and add server-side controls if enforcement is required |
| Commented-out rules appear to take effect | Comments are removed before rules are interpreted | Treat commented lines as documentation only and remove them if they cause confusion |
| Specific group seems to be ignored | User-agent product tokens are matched without regard to letter case, so a token case mismatch in the file is not the cause | Confirm the token is exactly one of the 32 in the checker and that the path begins with a slash |
Run a Diagnostic Check in the AI Bot Robots.txt Checker
Troubleshooting works best when the inputs to the checker are the production file and the exact path a crawler would request. The three verified operating steps below give you a deterministic starting point; once you have a clean baseline, you can change one variable at a time to isolate the cause of an unexpected row.
- Paste the exact robots.txt text you want to review and keep it under the 500 KiB limit. Use the file as it will be served in production rather than a working draft, and remove any commented-out rules you do not want interpreted as live policy.
- Enter a case-sensitive URL path beginning with a slash, then run the check. Include the query string only when that is part of the crawler request you want to model, because the path is matched from its beginning against every Allow and Disallow pattern in the applicable group.
- Review every matched rule and verify critical product tokens against current operator documentation before deployment. Cross-check the report against the AI Bot Robots.txt Checker table and against the operator pages listed in the source status column.
When the first check produces a confusing row, change only the path and rerun. If a second path returns the answer you expected, the parser is working correctly and the original path was the variable. If the answer still looks wrong, swap the file for the production version you actually serve and rerun the same path before changing any other variable.
Why the Result Can Look Wrong: Matching Precedence
RFC 9309 defines a precise order of operations, and the checker follows it exactly. The parser reads up to 512,000 UTF-8 bytes, groups rules by user-agent, merges duplicate case-insensitive product-token groups, and falls back to the wildcard group only when no specific group matches. Within the applicable group, Allow and Disallow patterns are compared against the path from its beginning, the longest match wins, and Allow beats Disallow when both match with equal length. An asterisk matches any sequence of characters, and a dollar sign at the end anchors a pattern to the end of the path.
Three consequences follow from those rules. First, a matching product-token group takes precedence over the wildcard group, so an answer produced under User-agent: * is the fallback answer, not the primary one for that token. Second, when equally specific Allow and Disallow rules both match, Allow wins, which means a narrow Allow for a subdirectory can reopen a path that a broader Disallow otherwise closes. Third, an empty Disallow value does not block anything; it is treated as a present-but-empty rule that contributes zero match length.
If no applicable Allow or Disallow rule matches, the checker reports access as allowed. That default is consistent with RFC 9309 but is often the source of "I never allowed that token" complaints, because the file did not actively disallow it and the parser had nothing to compare the path against.
After the Check: Verifying the Live File
A passing report on the pasted file is necessary but not sufficient. Once you publish the robots.txt, fetch the live /robots.txt from the exact scheme and host you intend, confirm the response is plain text and returns successfully, and verify behavior in available operator tools or server logs. The checker cannot perform any of those steps because it does not fetch URLs and does not contact crawler operators, which is also why it stays private and reproducible.
Common post-publication problems that the report cannot detect include a CDN serving a stale cached copy, a load balancer rewriting the response, a redirect that strips the file path, or a host-scope error that serves one site's rules to a different hostname. Each of these will leave a perfectly correct pasted-file report looking wrong on the live site, and each one has to be checked at the network layer rather than in the browser.
For a deeper look at the gap between a protocol decision and what actually happens on the wire, the guide on whether a robots.txt block actually stops AI crawlers walks through the same separation between the file and enforcement.
When to Look Beyond the Report
Robots.txt is voluntary. It is not authentication, authorization, a firewall, a contractual enforcement system, or proof that content will stay out of an AI model. A crawler can ignore the file, and publicly listing a path can reveal it to anyone who reads the same file. When the goal is enforceable protection of confidential or paid content, the right tools are server-side access controls, network-layer blocking, and traffic monitoring, not a more aggressive robots.txt.
An allowed row in the report means only that the pasted rules do not request a block for the selected path under the implemented protocol logic; it does not prove that the operator will crawl, index, cite, train on, or display the page. A blocked row likewise does not prove enforcement or removal from an index. Search indexing, snippet controls, training preferences, and live network blocking are separate controls maintained by each operator, and they are documented on the operator pages that the checker's source status column points to. Treat the report as a pre-deployment review, verify a critical policy against the operator's latest documentation, and then monitor actual traffic separately to confirm the policy you wanted is the policy you got.
Related reading: Is a Meta Robots Generator Safe? Pre-Use Checks That Matter.