A robots.txt file is a plain-text instruction file placed at the top level of a website that tells compliant crawlers which paths they may or may not request, and the entire file is generated in your browser in three steps: enter the site origin, choose a crawler policy, and publish the resulting UTF-8 text as /robots.txt. The file follows RFC 9309, the Robots Exclusion Protocol, which standardizes a single User-agent record followed by Allow or Disallow rules that match against the URL path from its first character. Because the rules are public, portable, and tiny, generating one for any HTTP or HTTPS website is a focused, deterministic task rather than a configuration project. The Robots.txt Generator handles the formatting: it normalizes the URL to its origin, accepts a single path pattern per line for selective blocking, and appends a Sitemap line that points to the default /sitemap.xml at the same origin. Everything stays in the current tab, so nothing is uploaded and no account is required. After generation you still own the publishing step — the file must be served at /robots.txt on the exact scheme, host, and port it is meant to govern.
What RFC 9309 defines in the output
RFC 9309 — the Robots Exclusion Protocol — defines only the structure of the file and the matching behavior that crawlers apply. The standard says a robots.txt file consists of one or more groups; each group begins with a User-agent line naming a product token or the wildcard *, followed by zero or more Allow or Disallow rules. A crawler with no more specific matching group falls back to the wildcard group. Rules are not commands; they are requests, and the standard explicitly states that they are not an access-control mechanism.
Two matching behaviors defined by the standard drive most of the surprises that show up in production files. First, paths are compared case-sensitively and from their first character, so /Private and /private are two different patterns. Second, when multiple rules in the same group could match a given URL, the most specific (longest-match) rule wins. That is why Disallow: /draft and Disallow: /draft$ do not necessarily express the same intent — the trailing dollar sign is an end anchor that limits the rule to the exact path.
Generate the file in five steps
- Enter the complete HTTP or HTTPS URL for the website origin (for example https://example.com or https://example.com:8443). The generator normalizes the URL to its scheme, host, and explicit port before generating.
- Choose an overall crawler policy: Allow all, Block all, or Selective disallow. Each option maps to a single recognizable output line.
- If you chose selective disallow, enter one path per line. Every non-empty rule must start with a slash, exact duplicates are removed in first-seen order, and the list is capped at 50 rules. Hash signs are rejected so a stray comment cannot split a rule.
- Generate the file and inspect the text. The output includes one wildcard User-agent group, the chosen policy line or lines, and a Sitemap line pointing to <origin>/sitemap.xml.
- Download the file, compare it with the current production /robots.txt, and publish the reviewed version only after you have preserved a copy of the old file for rollback.
What each policy writes
The generator deliberately produces one general group rather than pretending to manage every search engine's private extensions. Each of the three policies maps to a single recognizable line, which makes the file easy to read and to diff against an existing version.
| Policy | Output rule | Effect on compliant crawlers |
|---|---|---|
| Allow all | Allow: / | Crawlers may request any path on the origin |
| Block all | Disallow: / | Crawlers should not request any path on the origin |
| Selective disallow | One Disallow: line per entered path | Crawlers should avoid the listed paths; everything else remains reachable |
Every output file is plain UTF-8 text with the two required line types: a User-agent: * record and the rule lines that follow it. The Sitemap line is appended last because it is not part of any group and is read independently by crawlers.
Path rules that change the meaning
Because RFC 9309 defines specific matching behavior, the characters you type into a path field are interpreted in defined ways. Wildcard asterisks and the trailing dollar sign are preserved because the protocol defines those special matching characters. Reviewing the table below before publishing catches most accidental over- or under-blocking.
| Pattern in the file | How crawlers compare it | Example URLs that match |
|---|---|---|
| /draft | Prefix match against the URL path | /draft, /draft/notes, /drafts |
| /draft$ | Exact match only (end anchor) | /draft only |
| /Private | Prefix match, case-sensitive | /Private, /Private/2024; not /private |
| /*.pdf$ | Wildcard plus end anchor for any root-level PDF | /files/report.pdf |
The longest-match rule is the second source of common confusion. If a path could be matched by both a short and a longer rule in the same group, the longer rule wins. That is why a broad Disallow: / does not always behave the way people expect when more specific Allow rules are layered on top of it in production files written by hand.
The Sitemap line and origin handling
The generator appends a single Sitemap: line to every output file, derived from the normalized origin you entered. A URL that includes a page path, query string, or fragment is reduced to its origin so the default sitemap does not accidentally inherit an unrelated route. Explicit ports — for example https://example.com:8443 — are preserved because they belong to a distinct origin in the protocol's eyes.
If your actual sitemap lives at a different location (a CDN URL, a path other than /sitemap.xml, or a separate sitemap index), edit the generated Sitemap line before publishing. The generator does not crawl the site or guess at a sitemap location; the line it writes is a deliberate default that you should replace with the real URL when one exists.
Where and how to publish the file
A robots.txt file only governs the exact scheme, host, and port where it is served. The generator's output must be uploaded as /robots.txt at the top level of that origin, with a lowercase filename, and served with the text/plain MIME type. A file placed in a subdirectory, on another subdomain, or behind a different protocol does not govern the origin you intended.
This origin-scoped serving requirement is the most common reason a freshly generated file appears to do nothing. If the live site is on https://www.example.com and the file is uploaded to http://example.com/robots.txt, the HTTPS origin has no file and the HTTP origin has one — but crawlers fetch from the origin they were given, so they see an empty result where you expected instructions.
What robots.txt does not do
RFC 9309 states that the protocol is not an access-control, authorization, or security mechanism. Compliant crawlers are expected to honor the rules, but a malicious client can ignore them, and every listed path is publicly visible the moment the file is published. Treat a robots.txt file as a polite request to crawlers you want to cooperate with — not as login protection, encryption, or a privacy control.
Blocking crawling also does not guarantee that a URL disappears from search results. A search engine may discover a URL through inbound links and retain limited information about the page (such as its title or a snippet) without fetching the blocked body. For real removal, use the search engine's URL removal tools, the page-level X-Robots-Tag header or meta robots tag, or server-side authentication that prevents anonymous access in the first place. Crawling and indexing are separate steps, and the right control depends on the actual goal.
Before replacing a production file
The generator does not fetch the existing live file, validate server responses, submit the file to a search engine, or confirm that a crawler has refreshed its cache. Those actions require access to the deployed site and to search-engine tooling. Before replacing the production file, download the generator's output, compare it line-by-line with the current file, preserve any intentional crawler-specific groups the old file contained, and keep a copy of the old file for rollback.
Where the relevant search engine provides a tester — Google's robots.txt tester is the canonical example — point it at important public URLs to confirm they remain reachable, and at the disallowed paths to confirm they return a blocked status. For per-search-engine control beyond the wildcard group, edit the generated text directly. The standard defines the group structure, so additional User-agent records can be appended without breaking the file.
The generator intentionally does not add a Crawl-delay line because that directive is not part of RFC 9309 and is not universally supported. If your hosting platform or a particular crawler documents its own delay parameter, treat that as an out-of-protocol extension and add it manually rather than expecting the generator to produce it.
For a deeper look, see Meta Robots Generator for Beginners: Build Your First Tag.
For a deeper look, see Command Line vs Online ads.txt Generator: Which to Pick.