A robots.txt generator is a focused utility that takes one website origin and one policy choice, then produces a small, standards-aligned robots.txt file as plain UTF-8 text in the browser, without uploading any configuration data. The output follows the group-and-rule model standardized by RFC 9309: a single User-agent: * group is followed by either Allow: /, Disallow: /, or a list of selective Disallow lines, then a single Sitemap line whose URL is reduced to the origin plus /sitemap.xml. Wildcard asterisks and trailing dollar signs are preserved because the protocol defines them as matching characters, and hash signs are rejected so a user-entered rule is never accidentally turned into a comment. Because the work happens entirely on the page, the generator never sees your live robots.txt, never submits anything to a search engine, and never validates server responses — those steps still require access to your deployed site and the relevant search-engine tools.
What a robots.txt generator actually produces
The robots.txt file is a tiny plain-text file placed at the top level of a site origin. It is the first thing compliant crawlers check before requesting any other resource, and it contains only voluntary instructions, not enforcement. The Robots.txt Generator creates this file entirely in the browser: you enter one HTTP or HTTPS URL for the website origin, choose a policy, and the tool emits a downloadable UTF-8 text file that respects the rules defined by RFC 9309.
The structure of the output is deliberately narrow. The generator emits exactly one group, beginning with User-agent: *. The asterisk is the wildcard product token, and it applies whenever a crawler has no more specific matching group. Under that single header, the tool writes either an Allow or Disallow rule (or a list of Disallow rules in selective mode), then appends Sitemap: followed by the normalized origin and /sitemap.xml. No additional headers, crawler-specific blocks, or non-standard directives are added, because the tool does not pretend to manage every search engine's private extensions. The tool keeps everything in the current browser tab: entered paths and generated text do not leave your device.
The three policy modes and what each emits
The generator offers three overall policies. Each maps to a specific set of lines, and the choice changes the file more than the choice of a single path. Allow all mode writes Allow: / under the user-agent group, which signals compliant crawlers that they may request any path on the origin. Block all mode writes Disallow: /, which asks compliant crawlers not to request any path; it is the simplest way to make a site temporarily invisible to compliant bots.
Selective mode accepts one path pattern per line and writes each one as its own Disallow rule. The generator requires every non-empty rule to begin with a slash, removes exact duplicates in first-seen order, and caps the list at 50 rules. Wildcard asterisks and a trailing dollar sign are kept because the protocol defines those characters as matching syntax; hash signs are rejected so a path can never become a comment. After the rule block, the tool appends Sitemap: followed by the normalized origin and /sitemap.xml. A URL that contains a page path, query, or fragment is reduced to its origin so the default sitemap does not accidentally inherit an unrelated route, and explicit ports are preserved.
| Policy mode | What the generator writes | Typical use |
|---|---|---|
| Allow all | Allow: / | Open sites, staging mirrors, or any origin that wants every compliant bot to crawl everything |
| Block all | Disallow: / | A development host, a temporary takedown, or a site that wants to be invisible to compliant crawlers |
| Selective | One Disallow: line per entered path | A live site that needs to hide specific areas such as /draft, /private, or /internal-search |
Path matching rules that change how a Disallow line works
Path matching starts at the first octet of the URL path, not at the full URL, and crawlers compare paths case-sensitively. RFC 9309 specifies that crawlers use the most specific matching rule, which means /Private and /private are different paths and should be listed separately if both are intended to be blocked. Disallow: /draft does not necessarily express the same intent as Disallow: /draft$: the first matches /draft, /drafts, /draft-list, and any other path beginning with /draft, while the dollar sign anchors the match to the exact string /draft.
The generator does not validate whether your rule logic matches your intent. It only enforces that each rule starts with a slash, that there are no comment characters, and that no exact duplicates appear. Reviewing capitalization, prefixes, wildcards, and anchors is the reader's responsibility before publishing. If there is any doubt about which paths a particular rule will match, run a representative sample of URLs through a search-engine tester rather than guessing.
How to generate a robots.txt file with the tool
- Enter the complete HTTP or HTTPS URL for the website origin (for example, https://www.example.com). Do not paste a page path, query, or fragment — those inputs are reduced to the origin.
- Choose the overall crawler policy. For Allow all, select the option that writes Allow: /. For Block all, select the option that writes Disallow: /. For selective mode, choose selective and prepare to type one path per line.
- In selective mode, type each path beginning with a slash. Wildcard asterisks and a trailing dollar sign are accepted; hash signs and rules without a leading slash are rejected. Keep the list to 50 unique entries.
- Generate the file and inspect the text. Confirm the User-agent: * line, the Allow or Disallow rules, and the Sitemap: line that points at the origin plus /sitemap.xml.
- Download the result, then compare it with the current production robots.txt. Keep a copy of the old file for rollback and preserve any intentional crawler-specific groups before publishing.
- Publish the result as /robots.txt at the top level of the exact scheme, host, and port it governs, served as text/plain.
Where the output must live to actually apply
A robots.txt file only governs the origin where it is published. The path is always /robots.txt, the filename is lowercase, and the MIME type should be text/plain. A file placed in a subdirectory does not govern the site. A file on a different subdomain does not govern the other subdomain. A file served over HTTP does not govern the HTTPS version of the same host, and vice versa.
Explicit ports matter: a file served from https://example.com:8443/robots.txt does not apply to https://example.com/. If your sitemap lives at a different location than the origin plus /sitemap.xml, edit the generated Sitemap: line before publishing. The generator only writes the default sitemap URL based on the origin you entered; any non-standard location is left for you to correct.
What robots.txt does not do
Robots.txt is not authentication, authorization, encryption, or a privacy control. RFC 9309 explicitly states that these rules are not access authorization: compliant crawlers may honor the requests, but a malicious client can ignore them, and every listed path is publicly visible. Never list a sensitive route as a substitute for login checks or server-side access control.
Blocking crawling also does not guarantee that a URL disappears from search results. A search engine may discover a URL through links and retain limited information without fetching the blocked page. For removal or privacy goals, use the appropriate page-level indexing controls, HTTP headers, removal workflows, or authentication. Robots.txt is a request, not a removal tool. The generator also does not write Crawl-delay because the directive is outside RFC 9309 and is not universally supported; omitting it is the standards-correct choice. The tool does not fetch the existing live file, validate server responses, submit the file to a search engine, or confirm that a crawler has refreshed its cache. Those steps require access to the deployed site and the relevant search-engine tools, such as the tester described in Google's robots.txt documentation.
Related reading: Ads.txt Generator on iPhone: Build It in Safari.
Related reading: What Can Go Wrong When You Check AI Bot Robots.txt.