What the 500 KiB Robots.txt Limit Means for AI Crawlers

The 500 KiB robots.txt limit means parsers must handle at least 512,000 bytes, while the AI Bot Robots.txt Checker accepts up to that much UTF-8 input and applies additional safety caps. RFC 9309 sets the minimum parsing capacity; it does not guarantee that every crawler will fetch that much, follow the file, or use the same operational limits. In practical terms, the ceiling determines how much policy text the checker can parse for an access report, not a page-size, crawl-rate, indexing, training, or security limit. The checker also caps input at 5,000 non-empty access rules and aggregate matching work at 20,000,000 operations. A file can therefore measure less than 512,000 bytes and still be rejected as a whole if it exceeds the rule-count limit or reaches the aggregate matching-work limit. Treat the submission as a deterministic review of the text and path provided, not as proof that the file is live, that a crawler fetched it, or that an operator will enforce the requested policy.

What happens beyond the 500 KiB boundary

A common interpretation is that a crawler honors the first 512,000 bytes and ignores everything after them. The requirement in RFC 9309, the Robots Exclusion Protocol, does not establish that universal behavior. It sets a minimum parsing capacity: a conforming parser must handle at least 500 KiB. An implementation may support more, and individual crawler products can have separate retrieval, caching, or operational controls. Their handling should therefore be confirmed from current documentation rather than inferred from this one boundary.

The checker has a separate, definite input boundary: it accepts no more than 500 KiB of UTF-8 robots.txt text. An over-limit paste is outside the accepted range, so do not expect a partial interpretation or partial access report. For this workflow, the size ceiling applies to the pasted file itself. The URL path is entered separately for matching and does not create additional room inside the file. Because the limit is byte-based, visible character count is not the acceptance test; review the actual UTF-8 byte size of the exact text you intend to deploy.

Limits that can reject a smaller file

The file-size ceiling is only one condition for completing a check. The protocol baseline, product input range, and browser safety limits serve different purposes:

Limit or coverage	Defined value	What it controls
RFC 9309 parser baseline	At least 500 KiB	The minimum amount a conforming robots.txt parser must support, not a universal truncation rule.
Checker file input	At most 500 KiB of UTF-8, or 512,000 bytes	The maximum accepted text for one pasted robots.txt file.
Access-rule input	5,000 non-empty access rules	The maximum rule count the checker accepts for a submission.
Matching work	20,000,000 aggregate operations	A safety ceiling for evaluating the rules against the entered path.
Product-token coverage	32 fixed robots.txt rows	The AI crawler, assistant, training-control, and AI-adjacent search tokens included in the report.

The matching-work figure is not another unit of file size and is not a crawler quota. It helps prevent an adversarial combination of a long path and a large rule set from freezing the browser’s main thread. If the rule-count limit is exceeded or the aggregate-work ceiling is reached, the checker rejects the file atomically rather than returning a partial report. That is why a file comfortably below 500 KiB can still fail to run.

Comments are removed before rules are interpreted, and lines other than User-agent, Allow, or Disallow do not affect this access report. They can still occupy bytes in the pasted file. Sitemap and Crawl-delay lines may be meaningful to particular crawlers even though they are outside this checker’s access-decision report, so they should not be removed merely to make the checker accept a file.

Check a production robots.txt file under the cap

1. Prepare the exact text. Paste the exact robots.txt text you want to review and keep it under the 500 KiB limit. If the intended production file is larger, reduce that file first rather than checking an arbitrary prefix or excerpt.
2. Enter the request path. Provide a case-sensitive URL path beginning with a slash, then run the check. You may include a query string when it is part of the crawler request you want to model.
3. Review the matched policy. Examine every matched rule and the allowed or blocked result for each of the 32 product tokens. Check specific-token groups separately from the wildcard group and confirm that the most specific matching pattern produces the displayed outcome.
4. Verify critical tokens before deployment. Compare important results with current operator documentation. The source labels help distinguish primary product documentation from cross-references and supplemental directory entries, which is important when crawler names, purposes, or policies change.

Interpret the report across 32 product tokens

The checker follows the core matching behavior standardized by RFC 9309. During review, keep these rules separate from the input-size limits:

• Product tokens are case-insensitive. Multiple groups naming the same token have their rules combined, while paths such as /Private and /private remain case-sensitive.
• A specific group beats the wildcard group. The User-agent: * group applies only when no specific product-token group matches.
• Patterns are evaluated from the beginning of the path. The most specific matching pattern wins. If equivalent Allow and Disallow patterns match with the same specificity, Allow wins.
• An asterisk matches any character sequence. A dollar sign at the end of a pattern anchors it to the end of the path.
• Comments are ignored. An empty Disallow value does not block anything.
• No match means allowed. If no applicable Allow or Disallow rule matches, the access decision is reported as allowed.

The fixed table covers 32 robots.txt product tokens used by AI crawlers, AI assistants, model-training controls, and AI-adjacent search crawlers. A row is not a promise that its name appears verbatim in an HTTP User-Agent header. Dedicated controls such as Google-Extended or Applebot-Extended can represent model-training or data-use preferences, while other rows represent retrieval crawlers or assistant products. The report keeps those distinctions visible through operator, purpose-category, and source-status labels.

Primary operator documentation is used where available for OpenAI and Anthropic products. Cloudflare’s AI bot and crawler traffic reference provides a current cross-reference for major products, while supplemental Cloudflare Radar directory tokens are labeled as directory-sourced rather than operator-first-party verified. Because these product categories can change, verify any critical deployment decision against the operator’s latest documentation.

An allowed result means only that the pasted rules do not request a block for the selected path under the checker’s implemented logic. It does not prove that an operator will crawl, index, cite, train on, or display the page. A blocked result does not prove enforcement, compliance, or removal from an index. Robots directives, search indexing, snippet controls, training preferences, and live network blocking are separate controls.

Reduce file size or matching work safely

If the checker rejects an oversized file or a submission that is within 500 KiB, identify the boundary that was reached before changing anything. The goal is to preserve the intended crawler policy rather than merely make the input fit.

• For the byte ceiling: remove comments or obsolete access rules that are no longer needed, then review the smaller exact production text. Confirm that every intended product group and path remains present.
• For duplicate groups: consolidate repeated, case-insensitive product-token groups carefully. The checker already combines duplicate groups, so preserving all relevant Allow and Disallow rules is more important than reducing line count for its own sake.
• For the rule-count ceiling: remove redundant or stale directives after checking important paths. Because the longest matching pattern controls the outcome, deleting a seemingly redundant rule can change access.
• For the matching-work ceiling: use a realistic required path and simplify unnecessarily costly patterns where policy allows it. Recheck all critical paths after the change.
• Do not test a truncated prefix. A partial file may omit later groups or rules and cannot reproduce the decision for the complete production policy.

Separate rules review from live enforcement

Robots.txt is voluntary. It is not authentication, authorization, a firewall, a contractual enforcement mechanism, or proof that content will stay out of an AI model. A crawler can ignore it, and publicly listing a path can reveal that path. Confidential or paid content still requires server-side access controls, authentication, network controls, and traffic monitoring.

The checker processes the pasted text in your browser. It does not fetch your website, upload the file, or contact a crawler operator. Its deterministic result is useful for a pre-deployment review, but it cannot detect redirects, CDN overrides, client-specific responses, an incorrect host scope, caching delays, an unreachable file, or what is currently deployed.

After publishing, complete the live verification separately:

• Fetch /robots.txt from the exact scheme and host used in production.
• Confirm that the response succeeds and contains the expected plain-text file.
• Compare the live text with the reviewed copy and check available operator tools or server logs for relevant behavior.

This division keeps the file-size review reproducible while avoiding an unsupported conclusion: passing the checker confirms how the accepted pasted text and selected path were evaluated, not how every AI crawler or downstream system will behave.