The AI Bot Robots.txt Checker evaluates one pasted robots.txt file against 32 current robots.txt product tokens used by AI crawlers, AI assistants, model-training controls, and AI-adjacent search crawlers. After you paste your file and enter a URL path beginning with a slash, the tool reports an allowed or blocked result for every token using the core matching rules standardized by RFC 9309. All processing happens in the browser: the file is not uploaded, no live site is fetched, and no crawler operator is contacted. User-agent product tokens are matched without regard to letter case; path patterns are matched from the start, with case sensitivity, asterisk wildcard support, and dollar sign end-of-path anchoring. When multiple groups target the same product token, their rules are combined; a matching product-token group takes precedence over the wildcard group, and within a group the most specific matching pattern wins, with Allow preferred over Disallow at equal length. The checker accepts at most 500 KiB of UTF-8 input and 5,000 non-empty access rules. Use it as a deterministic pre-deployment review that mirrors the protocol logic crawlers will read.
Robots.txt has become harder to audit in 2026 because the crawler landscape has multiplied. Beyond the traditional search engines, dozens of retrieval crawlers, AI-assistant fetchers, and dedicated training-control tokens now ship their own product strings. A site owner who only checks for GPTBot or ClaudeBot can quietly leave training-control tokens like Google-Extended or Applebot-Extended unaddressed, or accidentally blanket-block an assistant that pulls live pages for grounding. A purpose-built checker that enumerates 32 product tokens at once, applies protocol-correct matching, and labels each row's source status turns a fragile manual review into a single repeatable step.
What the AI Bot Robots.txt Checker evaluates
The AI Bot Robots.txt Checker parses at most 512,000 UTF-8 bytes of pasted robots.txt into RFC 9309 user-agent groups. Duplicate case-insensitive product-token groups are merged so a bot declared twice under different casing behaves as one group. When no specific group matches, the wildcard group (User-agent: *) applies. Within the applicable group, Allow and Disallow patterns are compared against the path from its beginning, with full pattern support for asterisk and the terminal dollar sign anchor. The most specific matching pattern wins, and on an equal-length tie Allow wins. Wildcard segments are precompiled and matched safely, identical applicable-group policies are cached, and aggregate work is capped so a long path combined with a large rule set cannot freeze the main thread.
Access rules are capped at 5,000 non-empty directives, and aggregate matching work is capped at 20,000,000 operations. A file that fits inside the 500 KiB byte limit can still be rejected atomically by a safety limit before matching begins. Lines that are not User-agent, Allow, or Disallow directives do not affect the result; Sitemap and Crawl-delay are deliberately outside the access-decision report because they govern crawl scheduling and discovery, not access.
| Category in the 32-token table | What the row represents | Source labeling |
|---|---|---|
| Dedicated training-control tokens | Operators publish a separate token such as Google-Extended or Applebot-Extended to express model-training or data-use preferences independently of retrieval | Operator-first-party documentation where available |
| Crawler and assistant retrieval products | Bots and assistants that fetch URLs to power AI products, including search-adjacent crawlers | Operator documentation, with Cloudflare's maintained AI crawler reference as cross-reference |
| Supplemental directory entries | Tokens surfaced from a cross-reference directory rather than operator-first-party documentation | Labeled as supplemental, not presented as first-party verified |
Run the checker on your pasted robots.txt
The three documented steps cover the full interaction. Treat them as a tight pre-deployment checklist rather than a casual one-off check.
- Paste the exact robots.txt text you want to review and keep it under the 500 KiB limit. Use the production file you intend to deploy, not a stripped-down example, because any rule that differs from production will produce a result that differs from production.
- Enter a case-sensitive URL path beginning with a slash, then run the check. Include a query string only when that is part of the crawler request you want to model, because path matching is anchored at the start and operates on the entire submitted string.
- Review every matched rule and verify critical product tokens against current operator documentation before deployment. A row's source status (operator-first-party, Cloudflare cross-reference, or supplemental directory) tells you how much extra verification to apply.
Read each row in the 32-token result table
Each row is one robots.txt product token, not necessarily a literal HTTP User-Agent header. The checker labels the operator, the purpose category, and the source status so you can distinguish a retrieval crawler from a training-control token, and a first-party-verified entry from a supplemental directory entry. The result column tells you whether the submitted path would be allowed or blocked under the implemented protocol logic, given the applicable group and the path-pattern comparison rules.
| Path-pattern element | Behavior under RFC 9309 matching |
|---|---|
| /private | Matches /private and any longer path that starts with /private |
| /private$ | Matches only /private; the dollar sign anchors the pattern to the end of the path |
| * inside a pattern | Matches any sequence of characters |
| /Private versus /private | Matching is case-sensitive for paths, so the two can produce different results |
| Empty Disallow: value | Does not block anything; treat it as a no-op |
| No matching rule in the applicable group | Access is reported as allowed |
When a product-token group and a wildcard group both apply, the matching product-token group takes precedence. When multiple groups declare the same product token, their Allow and Disallow rules are combined before matching. Comments are stripped before interpretation, so a comment that looks like a directive is harmless. These rules together are why a single file can produce 32 different outcomes for one path: each row is evaluated against the group that targets its token, falling back to the wildcard group only when no specific group applies.
Test the paths that matter before deployment
A single path test is rarely enough. Most production robots.txt files have sections that need different treatment: the home page that an assistant should be able to cite, an internal search endpoint that should never train on, a paid-content folder that should not even be fetched, and an archive that should be reachable for retrieval but blocked for training. Run the checker once per representative path and confirm that every relevant row resolves the way you expect. For each path, ask: which row is the retrieval crawler, which row is the training-control token, and does the wildcard group accidentally cover anything the specific group missed?
Path composition also matters. A pattern written as /news matches /news, /news/2026/01/, and /newsletter. A trailing dollar sign tightens that to an exact match. If you want to block a directory and nothing else beneath it, anchor the pattern; if you want to block the directory and every nested path, leave it unanchored. Query strings are matched as part of the path only when you actually include them in the submitted test path, which is useful when a crawler requests /search?q=... but unhelpful when you submit a path that the crawler would never request.
Cross-check critical tokens against operator documentation
The checker enforces the protocol logic; it does not guarantee that a particular operator will honor a particular token the way you expect. Two rows can share a category and still behave differently in production. OpenAI publishes guidance for allowing its web crawlers and a separate publishers and developers FAQ that clarifies how its crawler tokens relate to its training preferences. Anthropic documents its crawler controls through its support center. For a broader cross-reference, Cloudflare's maintained AI crawler and bot traffic reference and its managed robots.txt reference describe how major products are classified and how managed robots.txt files are produced. Treat these sources as your verification layer: when a row is operator-first-party, confirm against the operator's current page; when a row is supplemental directory-sourced, treat it as a starting point that needs operator confirmation before it becomes policy.
Crawler products change over time, so any policy you depend on should be re-verified periodically. The checker's row for a given token tells you what your file requests today; only the operator's documentation tells you what that token actually does tomorrow.
Limits the checker cannot see for you
Because the tool does not fetch URLs, it cannot detect redirects, CDN overrides, syntax served only to certain clients, incorrect host scope, caching delays, or an unreachable file. It also cannot confirm what is currently deployed. A blocked result is only a request for non-access: a crawler can ignore the file, and publicly listing a path can reveal it. To understand the enforcement gap in more detail, see the guide on whether a robots.txt block actually stops AI crawlers. An allowed result is only the absence of a block under the implemented protocol logic; it does not prove that the operator will crawl, index, cite, train on, or display the page. Robots.txt is voluntary, and it is not authentication, authorization, a firewall, or a contractual enforcement system. Protect confidential or paid content with server-side access controls and monitor actual traffic separately.
Search indexing, snippet controls, training preferences, and live network blocking are separate controls that the checker does not attempt to evaluate. After you publish, fetch the live /robots.txt from the exact scheme and host, confirm the response is plain text and returns successfully, and verify behavior in available operator tools or server logs. The checker gives you a deterministic, reproducible result from the same text and path; deployment verification closes the rest of the loop.