An AI bot robots.txt check is correct when the pasted file is parsed under RFC 9309, the path you enter is exactly what a real crawler would request, and each of the 32 product-token results is cross-checked against the operator's current documentation before deployment. The AI Bot Robots.txt Checker applies the protocol's match logic entirely in the browser, so the outcome is reproducible and does not depend on a live fetch. That makes "correctness" a matter of three things: what you paste, what path you test, and what you do with the report afterward. A clean run is not, by itself, proof of live behavior, but it is the strongest pre-publication signal you can get without touching production. The steps below turn a paste-and-click review into a small, repeatable audit: paste the exact production text, enter a path that mirrors a real crawler request, read every matched rule against RFC 9309's tie-breakers, verify any critical token with the operator's own documentation, and confirm the live response after deployment.
Most mistakes in robots.txt reviews for AI crawlers come from misreading the protocol's tie-breaker rules, treating paths as case-insensitive, mixing up the wildcard group with a specific product-token group, or assuming the 32 listed tokens are literal User-Agent header strings. This article walks through the exact matching logic the checker applies, the inputs that change the answer, and the verification steps that turn a paste-and-click review into a reliable pre-deployment audit.
Why correctness is harder for AI bot robots.txt checks than for classic crawlers
Classic robots.txt reviews usually focus on a single user agent, such as Googlebot. AI-era robots.txt reviews now span a mixed set of retrieval crawlers, assistant retrieval products, and dedicated training-control tokens, all governed by the same RFC 9309 matching rules but with different operator policies. A file that behaves predictably for one product can behave unpredictably for another if a single product-token group is missing or a wildcard group accidentally overrides it.
Correct checking is also harder because robots.txt is voluntary. A "blocked" row in the report only means the pasted rules request a block under the implemented protocol logic; it does not prove enforcement, removal from an index, or that an AI company will respect the request. A "allowed" row only means the file does not request a block for that product on that URL path. Treat the checker as a pre-deployment review of the rules, not a guarantee about what an operator will do.
What RFC 9309 matching the checker actually applies
RFC 9309 standardizes the Robots Exclusion Protocol. The checker follows those rules literally, so understanding them is the fastest path to reading the report correctly. The rules that most often change an AI bot robots.txt result are:
- User-agent product tokens are matched without regard to letter case. Writing GPTBot or gptbot targets the same group.
- A matching product-token group takes precedence over the wildcard group. When no specific group matches, the wildcard group applies.
- Allow and Disallow patterns are compared against the path from its beginning. Matching is case-sensitive for paths, so /Private and /private can produce different results.
- The most specific matching pattern wins. Specificity is measured by the length of the matching pattern, with * matching any sequence of characters and a trailing $ anchoring the pattern to the end of the path.
- When equally specific Allow and Disallow rules both match, Allow wins. This is why a long Allow can rescue a path that a long Disallow would otherwise block at the same specificity.
- Empty Disallow values do not block anything. Disallow with no path is treated as "no rule" rather than "block everything."
- If no applicable Allow or Disallow rule matches, access is reported as allowed. The default is allow, not block.
- Comments are removed before rules are interpreted. Lines starting with # do not affect the result.
These rules are not advisory. They are what the checker implements, and any deviation in your expectations will appear as a "wrong" result that is actually a protocol-correct result.
How to check AI bot robots.txt correctly
Running a correct check means giving the tool clean inputs and reading every matched rule against the operator's published behavior. Follow these steps in order.
- Paste the exact robots.txt text you want to review. Copy the production file verbatim. The checker accepts up to 500 KiB of UTF-8 input and 5,000 non-empty access rules; inputs within that byte size can still be rejected by an aggregate work safety limit, so paste the smallest faithful version of the file you can.
- Enter a case-sensitive URL path beginning with a slash. Use a path that mirrors what a real crawler would request, including any query string that is part of that request. The path is matched from its start, so /blog and /blog/ can return different results when rules are anchored or use trailing $.
- Run the check. The checker parses the file into user-agent groups, merges duplicate case-insensitive product-token groups, falls back to User-agent: * only when no specific group matches, then evaluates the path against each of the 32 product tokens.
- Read every matched rule before trusting the row. For each product token, identify the user-agent group that applied, the longest matching Allow and Disallow pattern, and which rule won. Do not skip the wildcard group's behavior just because a specific group matched.
- Verify critical product tokens against the operator's current documentation. Crawler products change over time. Treat any row that controls a high-stakes path as unconfirmed until it appears in the operator's own reference. OpenAI and Anthropic publish first-party documentation; Cloudflare maintains a cross-reference for many operators, and supplemental directory-sourced tokens are labeled accordingly in the report.
- Repeat for every important path. Run a second path that should be allowed, a third that should be blocked, and a fourth that targets a known edge case such as a trailing-slash canonicalization, a query string, or a deeply nested private area.
- Compare with what the live deployment serves. After publishing, fetch /robots.txt from the exact scheme and host, confirm a plain-text 200 response, and re-run the same checks against the response body.
This sequence keeps the result deterministic. Two reviewers pasting the same text and entering the same path will see the same row in every column, which is the cheapest correctness signal you can get.
Verifying results against operator documentation
The 32 rows in the report are robots.txt product tokens, not a promise that each string appears verbatim as an HTTP User-Agent header. Some operators publish a dedicated control token, such as Google-Extended or Applebot-Extended, specifically for model-training or data-use preferences; other rows represent retrieval products. Treating a dedicated training-control token as identical to a retrieval token is one of the most common ways an AI bot robots.txt review goes wrong.
Use the table's operator, purpose category, and source status to decide what "correct" means for that row. For OpenAI products, primary operator documentation should resolve any ambiguity. Anthropic publishes web crawler controls directly. For cross-operator questions, the Cloudflare AI crawler reference and managed robots.txt documentation are useful starting points, and any row that traces back to a directory source rather than operator-first-party documentation is flagged in the report so you can plan a follow-up check.
Pre-deployment checklist before you trust the report
Before you ship the file, walk through the table below and confirm each item. Treat any unchecked item as a blocker for that specific row, not for the whole file.
| Item | What to confirm | Where it shows up in the report |
|---|---|---|
| Production text matches pasted text | No trailing whitespace, no editor-added comments, no copy-paste line wrap changes | Source side: the exact file you pasted |
| Specific group exists for each critical token | A matching product-token group is present, not just User-agent: * | User-agent column |
| Wildcard group fallback behavior is intentional | You know which rows fall back to * and which ones you forgot to cover | Source status column |
| Path was entered exactly as a crawler would request it | Case-sensitive path, including any query string, beginning with / | Result column applies only to this path |
| Longest-match and Allow-wins tie-breakers were applied | The reported winning rule is the longest match, and Allow wins on equal length | Matched-rule details in each row |
| Critical tokens verified against operator docs | High-stakes tokens cross-checked with the operator's current page, not just a third-party summary | Source status column |
| Sitemap and Crawl-delay lines noted separately | These do not affect the access decision and were not confused with Allow or Disallow | Not in the report; confirm in source |
What the checker cannot tell you
The checker analyzes only pasted text and the entered path. It does not fetch your live site, so it cannot detect deployment problems, redirects, CDN overrides, syntax served only to certain clients, incorrect host scope, caching delays, or an unreachable file. It also cannot confirm what is currently deployed on your server. Those limitations keep the tool private and deterministic, but they mean a clean report is not, by itself, proof of live behavior.
Robots.txt is not authentication, authorization, a firewall, or proof that content will stay out of an AI model. A crawler can ignore the file, and publicly listing a path can reveal it. For confidential or paid content, use server-side access controls and monitor actual traffic separately. Search indexing, snippet controls, training preferences, and live network blocking are separate controls that live outside this report.
After the report: confirm the live deployment
Once the report looks right, the next step is to confirm what is actually being served. Fetch /robots.txt from the exact scheme and host that crawlers will see, confirm the response is plain text and returns successfully, then run the same paths through the checker again using the live response. This catches edits that were staged but never deployed, build-pipeline rewrites that changed line endings, and CDN edge overrides. For a structured walkthrough of how to read each row against operator documentation, see the verification walkthrough.
If a row in the report looks wrong, the most common cause is one of three things: the path was entered with a different case or trailing slash than the rule expects, a specific product-token group is missing so the wildcard group is silently being applied, or the operator has updated its token since the report's source list was last refreshed. Re-paste the file, re-test the path, and re-check the operator's current documentation before changing the production rules.
A correct AI bot robots.txt check is not a single paste-and-click. It is a small audit: paste the production text, test the path that matters, read the matched rule, verify the token against the operator's own page, and confirm the live deployment matches what you reviewed. That sequence is what makes the 32 product tokens in the AI Bot Robots.txt Checker useful as a pre-deployment review rather than a guess. For broader cross-operator context on how a block interacts with each operator's policy, the Cloudflare AI crawler reference is a useful cross-reference alongside the operator's own documentation.
If you're weighing options, Does llms.txt Replace robots.txt or sitemap.xml? Explained covers this in detail.