Skip to content
Lizely

AI Bot Robots.txt Checker

Check one robots.txt file against 32 AI and AI-adjacent crawler product tokens with RFC 9309 matching, entirely in your browser.

Privacy: your files never leave your device. All processing happens locally in your browser.

How to use

  1. 1.Paste the exact robots.txt text you want to review and keep it under the 500 KiB limit.
  2. 2.Enter a case-sensitive URL path beginning with a slash, then run the check.
  3. 3.Review every matched rule and verify critical product tokens against current operator documentation before deployment.

About AI Bot Robots.txt Checker

AI Bot Robots.txt Checker evaluates one pasted robots.txt file against 32 current robots.txt product tokens used by AI crawlers, AI assistants, model-training controls, and AI-adjacent search crawlers. Enter a URL path, run the check, and review an allowed or blocked result for every token. Processing stays in the browser; the tool does not fetch a website, upload the file, or contact a crawler operator.

The checker follows the core matching rules standardized by RFC 9309. User-agent product tokens are matched without regard to letter case. When multiple groups target the same product token, their rules are combined. A matching product-token group takes precedence over the wildcard group, while the wildcard group applies when no specific group matches.

Within the applicable group, Allow and Disallow patterns are compared against the path from its beginning. The most specific matching pattern wins. When equally specific Allow and Disallow rules both match, Allow wins. An asterisk matches any sequence of characters, and a dollar sign at the end anchors a pattern to the end of the path. Comments are removed before rules are interpreted.

The path should start with a slash and may include a query string when that is part of the crawler request you want to model. Matching is case-sensitive for paths, so /Private and /private can produce different results. Empty Disallow values do not block anything. If no applicable Allow or Disallow rule matches, access is reported as allowed.

The 32 rows are robots.txt product tokens, not a promise that every string appears verbatim as an HTTP User-Agent header. Some operators publish a dedicated control token, such as Google-Extended or Applebot-Extended, for model-training or data-use preferences. Other rows represent crawler or assistant retrieval products. The table labels each operator, purpose category, and source status so these distinctions remain visible.

Primary operator documentation is used where available for OpenAI and Anthropic products. Cloudflare’s maintained AI crawler reference and managed robots.txt documentation provide a current cross-reference for major products. Supplemental Cloudflare Radar directory tokens are labeled accordingly rather than presented as operator-first-party verified. Crawler products change over time, so verify a critical policy against the operator’s latest documentation before publishing it.

The checker accepts at most 500 KiB of UTF-8 input and 5,000 non-empty access rules. RFC 9309 requires parsers to handle at least 500 KiB, while this interactive browser tool also enforces a documented aggregate matching-work limit so an adversarial long path and large rule set cannot freeze the main thread. A file within the byte limit can therefore be rejected atomically by a safety limit. Lines that are not User-agent, Allow, or Disallow directives do not affect the result. Sitemap and Crawl-delay lines may be meaningful to particular crawlers, but they are outside this access-decision report.

Robots.txt is voluntary. It is not authentication, authorization, a firewall, a contractual enforcement system, or proof that content will stay out of an AI model. A crawler can ignore the file, and publicly listing a path can reveal it. Protect confidential or paid content with server-side access controls and monitor actual traffic separately.

An allowed result means only that the pasted rules do not request a block for the selected path under the implemented protocol logic. It does not prove that the operator will crawl, index, cite, train on, or display the page. A blocked result likewise does not prove enforcement or removal from an index. Search indexing, snippet controls, training preferences, and live network blocking are separate controls.

Use this tool as a pre-deployment review. Paste the exact production file, test several important paths, inspect specific and wildcard groups, and compare the report with operator documentation. After publishing, fetch the live /robots.txt from the exact scheme and host, confirm the response is plain text and returns successfully, then verify behavior in available operator tools or server logs.

Because this checker does not fetch URLs, it cannot detect redirects, CDN overrides, syntax served only to certain clients, incorrect host scope, caching delays, or an unreachable file. It also cannot confirm what is currently deployed. Those limitations keep the tool private and deterministic while making the result easy to reproduce from the same text and path.

Methodology & sources

Parse at most 512,000 UTF-8 bytes into RFC 9309 user-agent groups; merge duplicate case-insensitive product-token groups; fall back to User-agent: * only when no specific group matches; compare case-sensitive path patterns from the start with * and terminal $ support; select the longest match and prefer Allow on equal specificity; precompile safe wildcard segments; cap access rules at 5,000 and aggregate work at 20,000,000 operations; cache identical applicable-group policies; then evaluate the same path independently for a fixed 32-entry, source-labeled robots.txt product-token table.

Frequently asked questions

Does a blocked result prevent an AI company from accessing my content?
No. Robots.txt is a voluntary crawler request, not access control; use authentication, network controls, and monitoring for enforceable protection.
Are all 32 entries literal HTTP User-Agent header values?
No. They are robots.txt product tokens; some are dedicated control tokens, and supplemental directory-sourced entries are labeled rather than treated as first-party verified.
Why can a more specific Allow rule override Disallow?
RFC 9309 selects the longest matching path pattern and specifies that Allow wins when equivalent Allow and Disallow rules have the same match length.
Does this tool inspect my live website?
No. It analyzes only pasted text and the entered path in your browser, so it cannot detect deployment, redirect, caching, or host-scope problems.

Related tools

SEO & Webmaster guides

View all