The AI Bot Robots.txt Checker produces a deterministic, browser-side report that shows whether each of 32 current robots.txt product tokens — used by AI crawlers, AI assistants, model-training controls, and AI-adjacent search crawlers — is allowed or blocked for a single path you specify, evaluated strictly against the rules in the file you paste. The result is a 32-row table where every row carries an operator label, a purpose category, a source status, and a single allowed/blocked verdict derived from RFC 9309 matching logic. Because the tool does not fetch a website, upload the file, or contact a crawler operator, the report is fully reproducible from the same pasted text and the same entered path — but that also means the result only describes what your pasted rules request, not what your live server is serving or what any crawler will actually do. Reading the result correctly means understanding each of those limits before treating any row as final, and the steps below walk through exactly how to do that.
What the AI Bot Robots.txt Checker Reports
Running the check on the AI Bot Robots.txt Checker produces a single deterministic table with exactly 32 rows. Each row corresponds to a current robots.txt product token used by an AI crawler, AI assistant, model-training control, or AI-adjacent search crawler, and each row carries four pieces of information: the operator label, the purpose category, the source status, and a single verdict of "Allowed" or "Blocked" for the path you entered.
The purpose category tells you what the token actually controls. Some rows represent dedicated training-control tokens that operators publish so publishers can opt out of model training without blocking retrieval — examples include entries such as Google-Extended or Applebot-Extended. Other rows represent crawler or assistant retrieval products that fetch pages for AI experiences and AI-adjacent search. A small number of rows are supplemental entries sourced from the Cloudflare Radar bots directory rather than from operator-first-party documentation, and those rows are explicitly labeled in the source-status column so you can see they are cross-references rather than primary operator statements.
The source-status column is the part most readers skim past and should not. Rows flagged as operator-first-party are safe to act on with normal caution; rows flagged as supplemental or directory-sourced deserve a closer look against the operator's own current documentation before the verdict is treated as authoritative. For a deeper walkthrough of how to read each column and how to interpret unusual category combinations, the guide on decoding the AI robots.txt report across 32 product tokens pairs naturally with this verification workflow.
Run the Check and Read Each Row
The check itself is a three-step browser workflow. Following the steps in order keeps the result reproducible and avoids the most common reasons a verdict surprises a site owner.
- Paste the exact robots.txt text you want to review into the input area. Keep the file at or under the 500 KiB UTF-8 limit the tool enforces; a file inside that byte limit can still be rejected if it exceeds either of two documented safety caps — the 5,000 non-empty access-rule limit or the 20,000,000 aggregate matching-work operation limit — so a useful pre-check is to scan for runaway groups or duplicated blocks before pasting.
- Enter the exact case-sensitive URL path you want to model, beginning with a slash. If the crawler request you are simulating includes a query string, include it; otherwise leave it off. The same file can produce different results for /Private and /private, so letter case is part of the input, not a stylistic choice.
- Run the check and review every matched rule the report highlights for each product token, then verify any critical verdict against the operator's current documentation before you publish or change anything.
Reading each result row means looking at four things in order: the product token (which crawler or control the row represents), the purpose category (retrieval versus training-control versus supplemental directory), the source status (operator-first-party versus labeled supplemental), and the verdict (Allowed or Blocked). A row that shows "Allowed" only means your pasted rules do not request a block for that token against the entered path; it does not mean the operator will crawl, index, cite, train on, or display the page. A "Blocked" row only means your pasted rules request that the token not access the path; it does not mean the operator honors the request or that the content has been removed from any existing index.
Verify a Result Against the Rules You Pasted
When a result looks wrong, the fastest way to check is to trace the verdict back to the matching rules in your pasted file. The AI Bot Robots.txt Checker implements the core rules standardized by RFC 9309, so the trace follows a predictable order. User-agent product tokens are matched without regard to letter case, which means GPTBot and gptbot target the same group. When several groups target the same product token, their rules are merged. A group whose product token matches the row takes precedence over a wildcard group, and the wildcard group only applies when no specific group matches.
Within the applicable group, the tool compares Allow and Disallow patterns against your path from the beginning. The longest matching pattern wins, so a more specific rule beats a shorter one that also matches. When equally specific Allow and Disallow rules both match the same path, Allow wins. An asterisk in a pattern matches any sequence of characters, and a dollar sign at the end anchors the pattern to the end of the path. Comments are stripped before any rule is interpreted, so anything that does not start with User-agent, Allow, or Disallow does not affect the verdict. Sitemap and Crawl-delay lines are visible in the file but are outside the access-decision report.
A short illustration of the matching logic uses a hypothetical group containing Disallow: /articles/2026/ and Allow: /articles/2026/review/. Against the path /articles/2026/review/, both patterns match because the Disallow pattern is a prefix of the path. The Allow pattern is longer, so it wins; the verdict is Allowed. If you instead tested /articles/2026/ itself, only the Disallow pattern matches, the length tie-break does not apply, and the verdict is Blocked. Empty Disallow values do not block anything, and if no applicable Allow or Disallow rule matches, access is reported as Allowed.
Cross-Check Critical Tokens With Operator Documentation
Robots.txt product tokens change over time. Operators rename crawlers, retire old ones, publish dedicated training-control tokens alongside their retrieval crawlers, and occasionally update which paths their products honor. The report is built from current operator documentation where it is available — primary operator documentation is used for OpenAI and Anthropic products, and the Cloudflare AI crawler reference plus its managed robots.txt documentation provide a maintained cross-reference for major products. Where a row comes from the Cloudflare Radar bots directory instead of an operator, the row is labeled accordingly rather than presented as first-party verified.
Two practical cross-checks cover most questions. For OpenAI products, the OpenAI publishers and developers FAQ describes which retrievers and which training-control tokens currently exist and how they should appear in robots.txt. For broader coverage of major AI and search crawlers, the Cloudflare AI bot and crawler traffic reference is a useful maintained cross-reference, alongside its managed robots.txt reference; for Anthropic products, the Anthropic web crawler controls article documents ClaudeBot and related tokens.
The cross-check matters most for training-control tokens, because they are the easiest category to misconfigure. A Disallow rule aimed at a retrieval crawler does not by itself opt a publisher out of training; only the operator-published control token does. If your goal is to keep pages out of training while still allowing retrieval, the report needs to be checked row by row against the operator's current control-token name.
After the Check: Confirm What Is Actually Deployed
The check analyzes only the text you paste. It does not fetch your website, upload the file, or contact any crawler operator, which is why the result is private and deterministic but also why it cannot on its own confirm what is live. Several deployment realities can make the on-page result diverge from the report, and a short verification loop catches each of them.
| Verification step | Why it matters |
|---|---|
| Fetch the live /robots.txt from the exact scheme and host | The report cannot detect redirects, CDN overrides, or rules served only to certain clients. |
| Confirm the response is plain text and returns successfully | A 200 status with a non-text content type, or a cached stale copy, can quietly change the effective file. |
| Compare the live file to the one you pasted into the checker | Caching delays or build-pipeline drift can leave a stale file deployed. |
| Verify the path scope is correct | The checker cannot confirm that the right virtual host or subdomain served the file you intended. |
| Confirm behavior in server logs or operator tools | A block is a voluntary request, not enforcement; only traffic evidence shows what actually happened. |
For a focused answer to whether the checker inspects your live site — and what that means for the trust you can place in the result — the guide on whether the AI Bot Robots.txt Checker fetches your live website walks through the privacy and scope boundaries in more detail. Treat the checker as a pre-deployment review: paste the exact production file, test several important paths rather than one, inspect both specific and wildcard groups in your file, and only after publishing the live file does it make sense to confirm the behavior you expected.
The same principle applies to the broader question of enforcement. A "Blocked" result in the report is a request, not a wall; a crawler can ignore the file, and publicly listing a path in Disallow can even reveal the path it is meant to protect. Confidential or paid content should sit behind server-side authentication and network controls, and actual crawler traffic should be monitored in logs. Used that way, the 32-row report becomes a fast, reproducible pre-publish review rather than a final answer.