When you check an AI bot robots.txt file, the result for any of the 32 product tokens is determined by four details: the case-insensitive User-agent group your rule sits in, the case-sensitive path you enter, the longest-matching Allow or Disallow pattern under RFC 9309, and whether the path begins with a slash. Get any one of those four wrong and every row in the report shifts, regardless of how carefully the rest of the file was written.
The AI Bot Robots.txt Checker reads one pasted robots.txt file against 32 documented product tokens used by AI crawlers, AI assistants, model-training controls, and AI-adjacent search crawlers. Every detail that matters flows from RFC 9309's standardized matching rules, but the checker also surfaces a per-token source label that distinguishes first-party operator documents from directory-sourced references. Whether a row appears as "allowed" or "blocked" is a property of the pasted text plus the path you typed, never of the live website. That separation lets you iterate on the policy privately before any file is published, but it also means the most consequential details are the rules you write, the paths you test, and the operator documentation you compare against after the report comes back.
The 32 Tokens and Why They Aren't All the Same User-Agent
The 32 rows in the report are robots.txt product tokens, not a promise that every entry appears verbatim as an HTTP User-Agent header. Some operators publish a dedicated control token — examples include Google-Extended and Applebot-Extended — to express model-training or data-use preferences independent of retrieval. Other rows represent crawler or assistant retrieval products that do send their own User-Agent string. The table labels each operator, purpose category, and source status so the difference stays visible. Primary operator documentation is used where available for OpenAI and Anthropic products, with the source label pointing readers to OpenAI's publishers FAQ and to Anthropic's crawler controls documentation. Supplemental Cloudflare Radar directory tokens are labeled accordingly rather than presented as operator-first-party verified. The source label on every row is itself a detail to read, because decisions about training data opt-outs depend on the dedicated control token and not the retrieval crawler token.
Path Matching Details That Drive Every Result
Five path-level details quietly decide the outcome for each token. Paths are matched case-sensitively, so /Private and /private can produce different rows. The path you enter should start with a slash. A query string can be appended to model a crawler request, but only when that string is part of the request you actually receive. Within the applicable group, Allow and Disallow patterns are compared from the beginning of the path. An asterisk matches any sequence of characters, and a dollar sign at the end of a pattern anchors it to the end of the path. Empty Disallow values do not block anything; they are records that the operator chooses to publish no block. Comments are removed before the rules are interpreted, so a comment that looks like a block is never a block. If no applicable Allow or Disallow rule matches, access is reported as allowed — which is itself a detail worth noticing, because the default is "allowed" rather than "blocked".
Run a Check With the AI Bot Robots.txt Checker
- Paste the exact robots.txt text you want to review and keep the input under the 500 KiB limit. The 500 KiB ceiling matches the minimum RFC 9309 requires parsers to handle, so the file must already be deployable on a single scheme-and-host origin.
- Enter a case-sensitive URL path that begins with a slash, appending a query string only when the request you want to model includes one, then run the check.
- Review every matched rule. For critical product tokens, compare the row against the operator's current documentation before you publish the file. The checker does not fetch the site or contact the operator, so verification is still your responsibility.
Rule Specificity, Precedence, and Source Labels to Verify
Two robots.txt fields decide ordering: the User-agent product-token group and the Allow or Disallow pattern. User-agent product tokens are matched without regard to letter case, so GPTBot, gptbot, and GPTBOT resolve to the same group. When multiple groups target the same product token, their rules are combined. A matching product-token group takes precedence over the wildcard group, while the wildcard group applies only when no specific group matches. Within the applicable group, the longest matching pattern wins, and when an Allow and a Disallow rule have the same match length, Allow wins per RFC 9309. The combined effect means a single line can change dozens of rows in the report, which is why the matching engine is also the most consequential detail to test against real paths you care about.
| Detail | Behavior |
|---|---|
| User-agent product token comparison | Case-insensitive |
| Path comparison | Case-sensitive, evaluated from the start |
| Asterisk (*) | Matches any character sequence |
| Dollar sign ($) | Anchors a pattern to the end of the path |
| Group precedence | Specific product-token group beats the wildcard group |
| Tie-breaker for equal-length matches | Allow wins over Disallow |
| Comment lines | Removed before interpretation |
| Empty Disallow value | Blocks nothing |
| Default with no matching rule | Reported as allowed |
The checker enforces a documented aggregate matching-work limit so that a long adversarial path combined with a large rule set cannot freeze the main thread; a file within the 500 KiB byte limit can still be rejected by this safety check, along with any input containing more than 5,000 non-empty access rules. Lines that are not User-agent, Allow, or Disallow directives — including Sitemap and Crawl-delay — do not affect the access decision, even though those lines may still be meaningful to particular crawlers outside the scope of this report.
Details the Checker Cannot See: Live Deployment and Behaviour
A pre-deployment review only covers the file as written. After publishing, fetch the live /robots.txt from the exact scheme and host, confirm the response is plain text and returns successfully, then verify behaviour in available operator tools and server logs. The checker never fetches URLs, so it cannot detect redirects, CDN overrides, syntax served only to certain clients, incorrect host scope, caching delays, or an unreachable file. It also cannot confirm what is currently deployed. Those limits keep the tool private and deterministic while making the result easy to reproduce from the same text and path. Reading the report as a property of the file rather than a property of the network is the detail that holds the rest of the workflow together.
A separate, larger detail deserves attention before any policy is shipped: robots.txt is voluntary. It is not authentication, authorization, a firewall, or a contractual enforcement system, and a crawler can ignore the file. Publicly listing a path can also reveal it. The checker reports only whether the pasted rules request a block for the chosen path under the implemented protocol logic; an allowed result does not prove that an operator will crawl, index, cite, train on, or display the page, and a blocked result does not prove enforcement or removal from an index. For a clearer picture of how a block relates to actual crawler behaviour, see Does a Robots.txt Block Actually Stop AI Crawlers?. Protect confidential or paid content with server-side access controls and monitor actual traffic separately — that is the only layer that delivers enforceable protection.
For a deeper look, see Details That Matter When You Use an HTML Page Weight Analyzer.