A blocked result from an AI crawler robots.txt check is a request, not a fence. Under the standardized rules in RFC 9309, robots.txt is a voluntary file that crawlers may ignore, so reporting that a path is blocked for a token like GPTBot or ClaudeBot only describes what your file asks the operator to do, not what the operator will actually honor. The same applies to dedicated training-control tokens such as Google-Extended or Applebot-Extended: a Disallow rule signals a preference, but it does not authenticate, authorize, or block at the network layer. To protect content from being seen, fetched, or used for training, you need server-side access controls such as login walls, paywalls, or IP blocks, not just a robots.txt rule. Cloudflare's AI crawler reference and managed robots.txt documentation both describe this same voluntary model, and the AI Bot Robots.txt Checker applies the same RFC 9309 logic in your browser against the operators it lists. That distinction is the single most important thing to understand before you rely on a "blocked" label in any robots.txt report, because a clean report of disallow rules is a planning artifact, not a guarantee of enforcement.
What a Blocked Result Really Means (and Doesn't)
When you run a robots.txt check and see a "blocked" label next to a token like ClaudeBot or GPTBot, the report is telling you that your pasted rules contain a Disallow pattern that matches the path you entered under RFC 9309 logic. It is not telling you that the operator cannot or will not fetch the URL. Three distinctions matter once you start reading the table:
- Scope of the check. The AI Bot Robots.txt Checker evaluates one pasted robots.txt file against 32 documented AI and AI-adjacent product tokens, applying the core matching rules standardized by RFC 9309. It does not call any operator, inspect any live deployment, or touch the network from your browser.
- What "blocked" actually says. A blocked result means only that the rules you pasted do not request access for that token under the implemented protocol logic. It does not prove that the operator will respect the rule, that the page has been removed from any index, or that training data has been retroactively purged.
- What the file cannot do. Robots.txt is voluntary. It is not authentication, authorization, a firewall, or a contractual enforcement system. A crawler can ignore the file, and publicly listing a path in a Disallow line can even reveal it to anyone who reads the file.
The table below summarizes what a single run can tell you and what falls outside its scope, so the difference between a request and a guarantee stays visible at a glance.
| The report tells you | The report cannot tell you |
|---|---|
| Whether your pasted rules contain a matching Allow or Disallow pattern for each of the 32 tokens | Whether the live /robots.txt is the file you pasted (deployment, redirect, cache, or host scope) |
| Which RFC 9309 group (specific product token vs wildcard) is the applicable group for a given token | Whether the operator honors the rule, fetches the URL anyway, or uses the page for training |
| The longest matching pattern with asterisk and dollar sign anchors, with Allow winning on ties | Whether the page is already indexed, cited, or present in a training corpus |
| That empty Disallow values do not block, and empty groups fall back to User-agent: asterisk | Whether server-side access controls (login, paywall, IP block) are actually in place |
| Source status for each row: first-party, managed, or supplemental directory | Whether other access paths (APIs, partner feeds, search engine caches) still expose the content |
How to Test Your robots.txt Against 32 AI Product Tokens
- Open the AI Bot Robots.txt Checker in your browser. Paste the exact robots.txt text you want to review, keeping the file under the 500 KiB UTF-8 limit the tool enforces.
- Enter a single case-sensitive URL path beginning with a slash. The path may include a query string when that is part of the crawler request you want to model. Matching is case-sensitive, so /Private and /private can produce different results even when the rest of the rule is identical.
- Run the check. The tool parses the file into RFC 9309 user-agent groups, merges case-insensitive duplicate product-token groups, falls back to User-agent: asterisk only when no specific group matches, and selects the longest matching Allow or Disallow pattern with Allow winning on equal length.
- Review the 32-row table. Each row reports allowed or blocked for the same path, labeled with the operator, purpose category, and source status so first-party crawler, dedicated training-control, and supplemental directory tokens remain visible.
- Cross-check critical product tokens against current operator documentation before you publish. Crawler products change over time, and the report explicitly labels supplemental directory entries as such rather than treating them as operator-first-party verified.
- After deployment, fetch the live /robots.txt from the exact scheme and host, confirm the response is plain text and returns successfully, then verify behavior in available operator tools or your server logs.
Because the tool does not fetch URLs, it cannot detect redirects, CDN overrides, syntax served only to certain clients, incorrect host scope, caching delays, or an unreachable file. Those limitations keep the check private and deterministic while making the result easy to reproduce from the same text and path.
What the 32 Product Tokens Represent
The 32 rows in the report are robots.txt product tokens, not a promise that every string appears verbatim as an HTTP User-Agent header. Some operators publish a dedicated control token, such as Google-Extended or Applebot-Extended, for model-training or data-use preferences that sit alongside their retrieval crawlers. Other rows represent crawler or assistant retrieval products that decide what the operator fetches in the first place. The table groups each row by purpose so those distinctions stay visible at a glance:
- Dedicated training-control tokens. These signal a data-use preference rather than a fetch preference. Blocking them is the closest the voluntary robots.txt system comes to opting out of training, and even that is still just a request.
- AI crawler retrieval tokens. These describe the actual retrieval product that fetches pages on behalf of the operator. Disallowing them reduces fetch attempts but, again, only as a request honored at the operator's discretion.
- AI assistant and search-adjacent tokens. These cover assistants and search-style products that combine retrieval with summarization or citation. Their behavior varies by operator, so each row is labeled with its operator and source status.
- Supplemental directory entries. Where primary operator documentation is unavailable, the checker uses Cloudflare's managed robots.txt reference and Radar directory tokens, and labels them accordingly rather than presenting them as operator-first-party verified.
Limits That Affect Whether the Check Can Run
The checker accepts at most 500 KiB of UTF-8 input and 5,000 non-empty access rules. RFC 9309 requires parsers to handle at least 500 KiB, and this interactive browser tool also enforces a documented aggregate matching-work limit so an adversarial long path and large rule set cannot freeze the main thread. A file that fits within the byte limit can therefore be rejected atomically by a safety limit if its rules exceed the cap or push the matching engine past its work budget.
Two other limits affect interpretation rather than acceptance, and both are easy to miss in a quick read:
- Lines that are not User-agent, Allow, or Disallow directives do not affect the result. Sitemap and Crawl-delay lines may be meaningful to particular crawlers, but they sit outside the access-decision report, so a sitemap line never turns a blocked row into allowed.
- An empty Disallow value does not block anything. If no applicable Allow or Disallow rule matches, the path is reported as allowed, which is consistent with RFC 9309 but easy to misread as a green light handed to the operator.
Going Beyond robots.txt for Real AI Access Control
Because robots.txt is voluntary, a blocked result is a starting point, not a finished control. If the goal is enforceable protection, layer robots.txt with the controls that actually keep content off the wire:
- Authentication and authorization. Put the path behind a login, a paywall, or a membership check so the page never returns content to anonymous requests, including crawler requests that ignore robots.txt.
- Network and origin controls. Use IP allowlists, firewalls, or a CDN challenge to block specific operator ranges where the operator publishes them. Treat the robots.txt rule as a polite request on top of a hard rule.
- Monitoring and review. Inspect server logs and referrer reports for the operators you care about. A rule you never measure is a rule you cannot verify, and a quiet server log is the only real proof that a block held.
- Index and snippet controls. Use meta robots tags, X-Robots-Tag headers, and any operator-specific preference pages to control search snippets, snippet usage, and dedicated training preferences separately from the fetch decision.
Used together, the AI Bot Robots.txt Checker becomes a pre-deployment review rather than a security tool. Paste the exact production file, test several important paths, inspect specific and wildcard groups, and compare the report with operator documentation before you publish. A blocked label on a token is the start of that workflow, not the end.