For large-text robots.txt work, this browser-based generator accepts up to 50 unique slash-prefixed paths in selective mode and outputs a downloadable UTF-8 file. It handles one website origin at a time; it does not import, merge, or automatically preserve an arbitrary existing robots.txt document. That distinction matters when the input is long: the tool creates a focused file with one general user-agent group, the selected rules, and a sitemap line.
The Robots.txt Generator creates that file entirely in your browser, so the site URL, paths, and generated text are not sent to a server and remain in the current tab. Enter the complete HTTP or HTTPS URL, and the tool reduces it to the origin for its output. A page path, query string, or fragment is discarded for this purpose, while an explicit port is preserved. The default sitemap reference becomes the normalized origin followed by /sitemap.xml, and you can edit that line if your actual sitemap is elsewhere.
Large-text limits to check first
RFC 9309 defines robots.txt as a collection of crawler groups, with each group beginning with User-agent and containing Allow or Disallow rules. The wildcard user-agent token applies when a crawler has no more specific matching group. This generator deliberately creates one User-agent: * group. It does not provide controls for named crawler groups, crawler-specific extensions, or private directives.
Selective mode accepts between 1 and 50 unique path patterns, with one pattern per line. If the current production file contains several crawler-specific groups, more than 50 selective patterns, or directives that must remain in place, treat the generated result as a draft rather than an automatic replacement. Download the existing file, compare it with the new version, preserve any intentional groups, and keep the old file available for rollback.
Choose the policy that matches the site
The three available policies produce different rule sets. Select the one that reflects your crawling goal, then review the generated text rather than assuming every directive will be interpreted identically by every crawler.
| Policy | Generated rule | Intended use | Important boundary |
|---|---|---|---|
| Allow all | Allow: / | Permit compliant crawlers to request every path. | It does not expose protected content or guarantee indexing. |
| Selective disallow | One Disallow line per selected pattern | Avoid listed paths while leaving other paths unblocked by this generated group. | The input is limited to 50 unique slash-prefixed paths. |
| Block all | Disallow: / | Ask compliant crawlers to avoid the entire origin. | Discovered URLs may still appear in search results. |
The policy determines the crawl request, not the site's security posture. Allow-all and block-all modes are easy to generate because each produces one main rule. Selective mode is more useful for a long list of routes, but its path count, syntax, and matching behavior still require review. A crawler may honor a permitted request without fetching every available URL, and a blocked request does not revoke public access to a resource.
Prepare a long path list
The selective input is a list of URL path patterns, not free-form text and not a complete existing robots.txt file. Each non-empty rule must start with a slash because path matching begins at the start of the URL path. If the list contains more than 50 paths, the generator cannot accept that full selection; identify the priority patterns or use a carefully reviewed wildcard rather than assuming every route is covered.
| Input detail | Generator behavior | Reason to review |
|---|---|---|
| Path count | Accepts 1 to 50 unique slash-prefixed paths. | Longer input lists must be shortened or consolidated. |
| Duplicate paths | Exact duplicates are removed in first-seen order. | Repeated entries do not create repeated rules. |
| Path prefix | Every non-empty rule must begin with /. | Rules are paths rather than complete site addresses. |
| Capitalization | Path comparisons are case-sensitive. | /Private and /private can refer to different paths. |
| Wildcards and anchors | Asterisks and a trailing dollar sign are preserved. | These are special matching characters and should be reviewed carefully. |
| Comments and control characters | Hash signs and control characters are rejected. | Remove them rather than expecting them to appear in the output. |
Review the list for capitalization, path prefixes, wildcards, and trailing anchors before generating the file. For example, /draft does not necessarily express the same intent as /draft$. The former can cover a longer path beginning with that text, while the latter anchors the match to the end. Use the narrowest rule that clearly expresses the intended request, then test important URLs after publication.
Generate and inspect the large path list
- Enter the complete website origin. Supply the full HTTP or HTTPS URL, including an explicit port when the site uses one. If you paste a page path, query, or fragment, the generator reduces the value to its origin for the default output.
- Choose an overall crawler policy. Select allow all, selective disallow, or block all according to the rules you intend to publish.
- Add the selective rules when needed. In selective mode, enter one path pattern per line. Use slash-prefixed paths, keep the list within 50 unique entries, and review capitalization, asterisks, trailing dollar signs, and any hash signs before continuing.
- Generate and inspect the text. Confirm that the file contains the expected User-agent group, the correct Allow or Disallow lines, and a sitemap URL based on the normalized origin. Edit the sitemap line if the sitemap is stored at another location.
- Download the result. Save the generated plain-text file. The generator creates it locally in the browser and does not send the entered configuration to a server.
- Compare it with the current production file. The tool does not fetch the live file, so obtain that file separately. Compare both versions, retain intentional crawler-specific groups, and save a copy of the old file for rollback.
- Publish the reviewed text at the correct location. Through your hosting or CMS controls, place it at /robots.txt on the exact scheme, host, and port it governs. Use the lowercase filename and serve it as text/plain.
- Test the live behavior. Check important public and blocked URLs with the relevant search-engine tester where available. The generator does not submit the file, validate the server response, or confirm that a crawler has refreshed its cache.
Read path matching before publishing
Under the standardized matching model, rule matching begins at the start of a URL path. Crawlers should compare paths case-sensitively and use the most specific matching rule. This means a broad prefix can be overridden by a more specific pattern, while paths that differ only by capitalization may remain separate. Preserve these details exactly when deciding what the long input list should contain.
The generated wildcard group applies to crawlers that do not have a more specific matching group already present in the file. If the production file includes named crawler groups, do not merge those directives mechanically. Preserve only groups you understand and intend to keep, because different crawlers can handle combined instructions differently.
- Check that each path begins with / and represents a route on the intended origin.
- Review whether capitalization matches the actual path exactly.
- Use a trailing dollar sign only when the match should end at that point.
- Confirm that each wildcard covers the intended routes rather than broader sections of the site.
- Compare every generated rule with intentional rules already in production.
Deploy the file at the origin it governs
A robots.txt file controls only the origin where it is published. The filename must be robots.txt, it must sit at the top level, and it must use the same scheme, host, and port as the site it is meant to govern. A file in a subdirectory, on another subdomain, or under a different protocol does not govern the intended origin. If the site runs on a non-default port, include that port consistently in the URL.
The default sitemap line points to /sitemap.xml on the normalized origin. Change it before publication if the real sitemap is elsewhere. For a Google property, consult Google's robots.txt creation guidance and test the live file after deployment. These checks are separate from generation because the tool does not inspect server responses or manage search-engine indexing.
Separate crawler control from security and indexing
robots.txt is a public crawler request, not authentication, authorization, encryption, or a privacy control. Every listed path is visible to anyone who can read the file, and a malicious client can ignore the instructions. Never list a sensitive route as a substitute for login checks or server-side access control. Use appropriate authentication and authorization when the objective is to prevent unauthorized access.
Blocking a crawl also does not guarantee that a URL will disappear from search results. A search engine may discover a URL through links and retain limited information without fetching the blocked page. Use page-level indexing controls, HTTP headers, removal workflows, or authentication according to the actual goal. The generator intentionally omits Crawl-delay because it is outside the RFC 9309 rules and is not supported consistently across crawlers.
Run a final production review
- Confirm that the URL represents the exact production scheme, host, and port.
- Verify that the selected policy matches the site's crawling objective.
- Review path capitalization, prefixes, wildcards, and trailing anchors.
- Check that the sitemap line points to the intended sitemap.
- Compare the generated file with the live file and preserve intentional groups.
- Keep the previous file available for rollback.
- Confirm that the live URL returns the file as text/plain.
- Test important public and blocked URLs with the relevant crawler tools.
If you're weighing options, What Is Schema Markup? Meaning, Types, and Limits covers this in detail.
If you're weighing options, Ads.txt Generator Tips and Common Mistakes covers this in detail.