One way to check AI bot robots.txt rules when evaluating robots.txt AI crawler access is to paste the exact production file into the AI Bot Robots.txt Checker, enter the request path, and review the allowed or blocked result for each of 32 documented AI and AI-adjacent crawler product tokens. The checker applies the core matching rules in RFC 9309 entirely in your browser, so it gives you a reproducible access decision without fetching a website or uploading the file. For the clearest result, use the same URL path you want to model, including a query string when relevant, then inspect every matched rule instead of treating a single Allow or Disallow label as the whole crawler policy. This approach is designed for pre-deployment review: it tests the text you supply against current product tokens, identifies the specific or wildcard group that governs each token, and separates protocol logic from operator behavior. It cannot prove that a crawler will obey the file, enter an index, train on a page, or display a citation, so current operator documentation and live-server checks remain essential.
Match the check to the request you care about
Start by identifying the exact path an AI crawler could request. A policy for a public article and a policy for an account area should not be tested with the same URL path. The checker evaluates the pasted text against the path you enter, so test the site root and each important public or protected path separately. Paths are case-sensitive: /Private and /private can produce different results. Include a query string when it is part of the crawler request you want to model.
The report’s 32 labels cover AI crawlers, assistant products, model-training controls, and AI-adjacent search crawlers. These are robots.txt product tokens, not a promise that every string appears verbatim in an HTTP User-Agent header. Some operators publish dedicated control tokens, such as Google-Extended or Applebot-Extended, while other entries represent crawler or assistant retrieval products. The table identifies each operator, purpose category, and source status. Primary operator documentation is used where available for OpenAI and Anthropic products. Cloudflare’s AI bot and crawler traffic reference provides a current cross-reference for major products, while supplemental Cloudflare Radar directory entries remain labeled as such rather than being presented as operator-first-party verification.
Check one robots.txt file in the browser
- Paste the exact robots.txt text you want to review. Keep the UTF-8 input no larger than 500 KiB, or 512,000 bytes, and use no more than 5,000 non-empty access rules. The file is parsed locally in your browser. A file within the byte limit can still be rejected atomically when the aggregate matching-work safety limit of 20,000,000 operations is reached; that is an input limit, not an Allow or Disallow result.
- Enter a case-sensitive URL path beginning with a slash. Model the path the crawler would request. Add a query string when it is part of that request. The checker compares the path from its beginning, so capitalization and query details can affect the outcome.
- Run the check and review every matched rule. Examine the result for all 32 product tokens, identify the specific or wildcard group responsible for each decision, and verify critical product tokens against current operator documentation before deployment.
The checker evaluates the same entered path independently for each of its 32 source-labeled token rows. Processing remains in the browser: it does not fetch a website, upload the file, or contact a crawler operator. The report therefore reproduces the implemented protocol decision from the exact text and path you supplied.
Read each result through RFC 9309 matching rules
Robots.txt evaluation depends on both group selection and path matching. The checker follows the core behavior standardized by RFC 9309, including the following defined outcomes.
| Rule situation | Checker behavior | What to inspect |
|---|---|---|
| A matching product-token group exists | The specific group takes precedence over the wildcard group. | Check every Allow and Disallow pattern in the specific group. |
| No specific group matches | The User-agent: * group applies when present. | Confirm that the wildcard group represents your intended fallback. |
| Several groups target the same product token | The applicable rules are combined. | Review all contributing directives rather than only the last one. |
| Patterns have different match lengths | The longest matching pattern wins. | Identify the most specific path pattern for the selected path. |
| Allow and Disallow match at equal specificity | Allow wins. | Look for equal-length rules that reverse an apparent block. |
| No applicable Allow or Disallow rule matches | Access is reported as allowed. | Do not mistake the absence of a match for an explicit Allow directive. |
| An applicable Disallow value is empty | It does not block anything. | Check for a meaningful path after the Disallow colon. |
User-agent product tokens are matched without regard to letter case, while URL paths remain case-sensitive. Within the applicable group, Allow and Disallow patterns are compared against the path from its beginning. An asterisk matches any sequence of characters, and a dollar sign at the end anchors a pattern to the end of the path. Comments are removed before rules are interpreted. Lines that are not User-agent, Allow, or Disallow directives do not affect this access-decision report.
Separate access decisions from crawler behavior
An allowed result means only that the pasted rules do not request a block for the selected path under the implemented protocol logic. It does not prove that an operator will crawl, index, cite, train on, or display the page. A blocked result likewise does not prove that access was prevented or that material has been removed from an index.
Robots.txt is a voluntary crawler request. It is not authentication, authorization, a firewall, a contractual enforcement system, or proof that content will stay out of an AI model. A crawler can ignore it. Protect confidential or paid material with server-side access controls and monitor actual traffic separately. Listing a confidential path in robots.txt can also reveal the path itself, so the file should not be treated as a private access list.
Search indexing, snippet controls, model-training preferences, and live network blocking are separate controls. A policy aimed at training may therefore need a different decision from one aimed at user-triggered retrieval. Review each token’s purpose rather than assuming that blocking one AI product blocks every assistant, crawler, or training system.
Verify the deployed file after publishing
After publication, fetch the live /robots.txt from the exact scheme and host you are serving. Confirm that the response is successful and contains plain text. Then verify behavior through available operator tools or server logs. The checker cannot detect redirects, CDN overrides, syntax served only to certain clients, an incorrect host scope, caching delays, an unreachable file, or what is currently deployed. For the boundary between a pasted-file test and a live-site test, see why the AI Bot Robots.txt Checker does not fetch your live website.
If the infrastructure uses a managed robots layer, account for that service separately and compare its behavior with Cloudflare’s managed robots.txt reference. A clean result from pasted text remains a protocol review, not proof of the response delivered to a particular crawler or client.
Turn the report into a documented policy decision
Use the report as a decision matrix. Identify each token’s purpose, note the matching specific or wildcard group, review the winning Allow or Disallow pattern, and compare the intended policy with current operator documentation. Test every path that represents a different access decision rather than relying on one broad path. Because crawler products and control tokens can change, verify a critical policy against the operator’s latest documentation before publishing it.
Finally, keep the reviewed file, tested paths, matched rules, and operator source associated with the deployment record. Then confirm the live file and inspect traffic independently. This sequence combines a reproducible robots.txt access check with the operational checks needed to manage real crawler requests without overstating what the file can guarantee.
For a deeper look, see How Do I Generate llms.txt? File Example and Checks.