The 32 entries in the AI Bot Robots.txt Checker are robots.txt product tokens, not literal HTTP User-Agent header values that a crawler sends with every fetch. A product token is the case-insensitive name that appears on a User-agent: line in robots.txt; it identifies a group of rules the parser should apply to that named crawler. The HTTP User-Agent header, by contrast, is whatever string a client chooses to advertise, and many crawlers and assistants send versions that include version numbers, contact URLs, or platform identifiers that are not part of the token used for matching. Treating those two concepts as the same leads to misconfigured rules and surprised teams, because a row in the table can represent either a retrieval crawler or a dedicated opt-out control token, and matching is done against the token, not the header string. Because of that, two different crawler products can share a near-identical header string while their robots.txt tokens behave as distinct groups, and a single operator can publish one token for fetching and a second token for opting out of training, both of which the checker evaluates against the same pasted file.
Some operators publish a separate product just for training-control purposes, with names like Google-Extended and Applebot-Extended, which appear alongside the retrieval crawler on the same host. Other operators expose one retrieval product token that the same client identifies through, and a few publish both. The AI Bot Robots.txt Checker evaluates one pasted robots.txt against the full set so that an allowed or blocked result covers both retrieval and training-control paths at once. When you open the checker, every row is labeled so you can tell whether a token corresponds to a retrieval crawler, an assistant, a model-training control, or an AI-adjacent search crawler, and whether the row came from primary operator documentation, a maintained cross-reference, or a supplemental directory.
Product Tokens vs. HTTP User-Agent Headers
A robots.txt file has no idea what an HTTP User-Agent header will look like. It only knows how to match the value that appears after "User-agent:" against the incoming crawler's product token. The protocol, defined in RFC 9309, deliberately treats the User-agent value as a case-insensitive product token so that any capitalization of "GPTBot" still resolves to the same group. The header a crawler actually sends is up to that crawler; some send the bare token, some append a slash and version, and some rotate identifiers entirely. Because of that gap, you cannot reliably block a crawler by typing the literal string it advertises in DevTools.
The practical effect is twofold. First, when you want to block a retrieval crawler, you address the product token such as GPTBot or ClaudeBot, not the string the operator shows in their published header samples. Second, when an operator publishes a separate training-control token like Google-Extended or Applebot-Extended, blocking the retrieval crawler does not block training use; only blocking the control token does. The two are independent opt-outs in the same file, and the checker shows both so neither is overlooked.
What Each of the 32 Rows Represents
The 32 rows are organized so you can read three things at a glance: which operator publishes the token, which purpose category it serves, and how the entry was sourced. The purpose categories cover AI crawlers, AI assistants, model-training controls, and AI-adjacent search crawlers. Source status spans primary operator documentation (used where available for OpenAI and Anthropic products), a maintained cross-reference such as Cloudflare's AI bot and crawler traffic reference, and supplemental directory entries from the Cloudflare Radar Bots Directory, which are labeled rather than treated as first-party verified.
| Purpose Category | Token Role | Typical Pattern |
|---|---|---|
| AI crawler | Retrieval product used to fetch pages into a model or feature pipeline | Operator name, often with a "Bot" suffix |
| AI assistant | Retrieval product tied to a conversational or in-product assistant | Operator-scoped assistant identifier |
| Model-training control | Dedicated opt-out token separate from the retrieval crawler | Operator-scoped name, often with an "Extended" suffix |
| AI-adjacent search | Search engine crawler or AI search product | Traditional search-bot name or branded AI search product |
The labels are the safety net for the central question. A row labeled as a model-training control is, by design, not the same string a crawler would put in an HTTP header; it is the rule group a publisher addresses to opt out of training. A row labeled as a search crawler is the token that a search engine's fetcher presents as a robots.txt product name. Both rows are evaluated with the same RFC 9309 matching logic, but the meaning of "allowed" or "blocked" for each depends on the category, which is why the column is shown.
Run the Checker Against Your Robots.txt File
The checker takes three inputs and returns one decision per token in the table.
- Paste the exact robots.txt text you want to review into the input area and keep it under the 500 KiB limit. Use the production rules rather than a sanitized draft, because group merging and rule combining happen during parsing and only the real file will surface the gaps you care about.
- Enter a case-sensitive URL path beginning with a slash, then run the check. The path may include a query string when that is part of the crawler request you want to model, and matching against the path is case-sensitive, so /Private and /private can produce different results.
- Review every matched rule in the report. Look first at tokens that control training use and at operators whose retrieval products you want to manage, then compare the result against the operator's current documentation before deployment.
You can repeat the check for several important paths. Treat each result as a pre-deployment review step rather than a guarantee; the same pasted file and the same path will always produce the same output, which makes the report easy to reproduce, share, and diff against operator documentation.
RFC 9309 Matching Logic Behind the 32 Rows
Under the hood, the checker parses up to 512,000 UTF-8 bytes into RFC 9309 user-agent groups, merges duplicate case-insensitive product-token groups, and falls back to the wildcard group only when no specific group matches. Within the applicable group, Allow and Disallow patterns are compared against the path from its beginning, and the most specific matching pattern wins. When equally specific Allow and Disallow rules both match, Allow wins. An asterisk matches any sequence of characters, and a dollar sign at the end anchors a pattern to the end of the path. Comments are removed before rules are interpreted, so a comment line never changes the outcome.
These rules apply identically to each of the 32 tokens, which is why a single pasted file can be evaluated across retrieval crawlers, assistant products, training-control tokens, and search crawlers in one pass. Empty Disallow values do not block anything. If no applicable Allow or Disallow rule matches, access is reported as allowed, which means the result is conservative: an "allowed" row simply means no rule blocks the path, not that the operator will fetch it.
Limits, Safety Caps, and What the Tool Does Not Do
The checker accepts at most 500 KiB of UTF-8 input and 5,000 non-empty access rules. RFC 9309 already requires parsers to handle at least 500 KiB, and this tool layers an additional aggregate matching-work cap of 20,000,000 operations so that an adversarial long path combined with a large rule set cannot freeze the main thread. A file within the byte limit can therefore be rejected by the safety cap before matching starts; the report will say so rather than run forever. Lines that are not User-agent, Allow, or Disallow directives do not affect the result, so Sitemap and Crawl-delay lines remain in the file for crawlers that read them but stay outside the access-decision table.
The checker does not fetch your website, does not upload the file, and does not contact a crawler operator. Processing stays in the browser. That keeps the report deterministic and reproducible, but it also means the tool cannot detect redirects, CDN overrides, syntax served only to certain clients, incorrect host scope, caching delays, or an unreachable file. It also cannot confirm what is currently deployed. After publishing, you still need to fetch the live /robots.txt from the exact scheme and host, confirm the response is plain text and returns successfully, and verify behavior in operator tools or server logs.
Verify Critical Tokens Against Operator Documentation
Crawler products change over time, and a row that was accurate when the checker was built may be out of date by the time you publish. Operators retire tokens, rename products, and add new ones. Treat the table as a snapshot and verify any policy you depend on against the operator's current documentation before you push it. Primary documentation is used where available for OpenAI and Anthropic products; Cloudflare's maintained AI crawler reference and managed robots.txt documentation, including the Cloudflare AI bot and crawler traffic reference, provide a current cross-reference for major products; and supplemental entries from the Cloudflare Radar Bots Directory are labeled rather than presented as operator-first-party verified.
For a high-stakes path, combine the checker with a live verification. Run the report, deploy the file, then fetch the live /robots.txt from the exact scheme and host, and check that the bytes match what you pasted. Inspect server logs for any of the 32 product tokens over the following days and confirm that an allowed token is actually fetching and a blocked token is actually staying away. Robots.txt is voluntary and is not authentication, authorization, a firewall, or proof that content will stay out of a model; protect confidential or paid content with server-side access controls and monitor actual traffic separately.