An AI bot robots.txt check should return an Allowed or Blocked result for each of 32 robots.txt product tokens for the exact path you test. This is a path-specific access decision calculated from the pasted file, not a promise that an AI operator will crawl, index, cite, train on, or display the page.
When checking robots.txt for AI crawlers, the useful expectation is a token-by-token report plus the rules that produced each decision. Results can differ by crawler product, URL path, and group structure. The same file can therefore produce different answers for public and restricted sections, while a correct report still says nothing about enforcement outside the robots.txt protocol.
What the result table contains
The checker evaluates one pasted robots.txt file against 32 current product tokens used by AI crawlers, AI assistants, model-training controls, and AI-adjacent search crawlers. Each token receives an Allowed or Blocked result for the path you enter, and you can review the rules that matched. Operator, purpose-category, and source-status labels keep different kinds of controls visible.
The 32 entries are robots.txt product tokens, not a guarantee that every string appears verbatim in an HTTP User-Agent header. Some operators publish a dedicated control token, such as Google-Extended or Applebot-Extended, for model-training or data-use preferences. See why all 32 entries are not necessarily literal User-Agent headers before comparing a row with an operator’s full header documentation.
| Reported outcome | What the report establishes | What it does not establish |
|---|---|---|
| Allowed | No applicable matching rule requests a block for the entered path under the implemented protocol logic. | That the operator will crawl, index, cite, train on, or display the page. |
| Blocked | A matching Disallow pattern wins the applicable-group comparison. | That a crawler will obey the rule, content will be removed, or protected data remains inaccessible. |
Source status is part of the interpretation. Primary operator documentation is used where available for OpenAI and Anthropic products. Cloudflare’s maintained AI crawler and bot reference provides a current cross-reference for major products, while Cloudflare Radar directory tokens are labeled as supplemental rather than treated as first-party verified. Verify any critical policy against the latest operator documentation because crawler products and controls change over time.
How the allowed or blocked decision is chosen
For each product token, the checker identifies the applicable user-agent groups and then compares their path patterns against the URL path you supplied.
- Product-token matching is case-insensitive. A token group is recognized regardless of letter case, while the URL path itself remains case-sensitive.
- Duplicate token groups are combined. When multiple groups target the same product token, their rules form the applicable policy.
- A specific group takes precedence over the wildcard. The User-agent: * group is used only when no specific product-token group matches.
- Patterns are compared from the beginning of the path. An asterisk matches any sequence of characters, while a terminal dollar sign anchors a pattern to the end of the path.
- The longest matching pattern wins. If equally specific Allow and Disallow patterns both match, Allow takes precedence.
- An empty Disallow does not block anything. When no applicable Allow or Disallow rule matches, access is reported as Allowed.
Comments are removed before rules are interpreted. Lines other than User-agent, Allow, and Disallow do not affect this access report. Sitemap and Crawl-delay directives may be meaningful to particular crawlers, but they fall outside the checker’s allowed-or-blocked decision.
Run the check on the text you intend to publish
Use the AI Bot Robots.txt Checker for a pre-deployment review of one pasted file. Processing stays in the browser: the tool does not fetch a website, upload the file, or contact a crawler operator.
- Paste the exact production robots.txt text. Keep the UTF-8 input under the 500 KiB limit and within the 5,000 non-empty access-rule limit. Even a file that meets those limits can be rejected atomically when its path-and-rule combination would exceed the checker’s 20,000,000-operation aggregate matching-work cap. That safety rejection is separate from an Allowed or Blocked crawler result.
- Enter the URL path you want to evaluate. The path must begin with a slash and is matched case-sensitively, so /Private and /private can produce different results. Include a query string only when it is part of the crawler request you want to model.
- Run the check and inspect all 32 rows. Review every matched rule, identify whether a specific product-token group or the wildcard group applies, and confirm which matching pattern determined the result.
- Verify critical product tokens against current operator documentation. Confirm that the token still represents the product and preference you intend to govern. Do not rely on a source-status label as a substitute for checking a policy that matters to your site.
- Confirm the published file separately. After deployment, fetch the live /robots.txt from the exact scheme and host, confirm that the response returns successfully as plain text, and verify behavior through available operator tools or server logs.
Read the result without treating it as enforcement
Robots.txt is a voluntary crawler request. A crawler can ignore it, and the file is not authentication, authorization, a firewall, a contractual enforcement system, or proof that content will stay out of an AI model. What a robots.txt block actually stops for AI crawlers explains why the protocol result should be interpreted within those boundaries.
Protect confidential or paid content with server-side access controls and monitor actual traffic separately. Publicly listing a restricted path can also reveal its existence. Search indexing, snippet controls, training preferences, and live network blocking are separate controls: an Allowed result does not prove that content will be used, while a Blocked result does not prove compliance or removal from an index.
Confirm the live file after publication
The browser checker analyzes only the pasted text and entered path. It cannot detect redirects, CDN overrides, syntax served only to certain clients, an incorrect host scope, caching delays, an unreachable file, or whether the reviewed file is currently deployed. A reproducible result from the pasted text therefore does not establish that the production URL serves the same policy.
After publication, retrieve the live file directly and compare it with the reviewed copy. When managed robots controls are part of your delivery stack, review that separate layer against Cloudflare’s managed robots.txt documentation, then use server logs or available operator reporting to check actual behavior.
See how one path can produce different results
Consider a wildcard group containing User-agent: * and Disallow: /private. For a token with no specific group, the path /private/report matches the block. If the same applicable group also contains Allow: /private/public-note, the path /private/public-note is Allowed because its matching pattern is longer. A token-specific group can produce a different result because it takes precedence over the wildcard group.
This illustrates why the expected answer is not one site-wide status: it is a 32-row, path-specific review supported by matched rules and source labels. Test representative paths, retain the exact file and path used for each review, and reproduce the same results from the same inputs. The final judgment should combine the protocol outcome, current operator documentation, and evidence from the deployed site rather than relying on one Allowed or Blocked label alone.