The Robots.txt Generator is a browser-based tool that takes a website URL, an overall crawler policy, and any paths you want to keep crawlers away from, then produces a plain UTF-8 text file you can publish as /robots.txt. The whole process runs locally in the current tab, so the URL and every path you type stay on your device and never reach a remote service. The output follows the group and rule model standardized by RFC 9309, with one wildcard User-agent, Allow or Disallow rules built from your inputs, and a Sitemap line pointing back at your site's origin plus /sitemap.xml.
People search for a free robots.txt generator for two overlapping reasons. The first is cost: writing the file by hand is free, but many site owners want a quick, structured starting point that already handles sitemap plumbing, slash prefixes, and rule order so they can focus on which paths to block. The second is control: a generator that runs entirely in the browser removes the privacy worry of pasting your site structure into a third-party form. Both motivations line up neatly with how the Robots.txt Generator is built — pick a policy, optionally list paths, and download the result.
Robots.txt Generator Modes Compared
Before you type a single path, choose the overall policy that describes how compliant crawlers should treat your site. The three modes map directly onto what RFC 9309 calls a "group of records": one User-agent line followed by rules a crawler must apply when no more specific group matches. The following table shows what each mode writes and when it is the right starting point.
| Mode | What it writes | Best starting point |
|---|---|---|
| Allow all | User-agent: *Allow: /Sitemap: https://example.com/sitemap.xml | Public sites with no sensitive folders and no temporary staging areas — you want every crawler to read everything. |
| Selective disallow | User-agent: *Disallow: /path1Disallow: /path2Sitemap: https://example.com/sitemap.xml | Most production sites — keep the bulk of the site crawlable while steering crawlers away from admin, search results, internal previews, or test areas. |
| Block all | User-agent: *Disallow: /Sitemap: https://example.com/sitemap.xml | Staging environments, URL shortener domains, throwaway microsites, or any origin you intend to take offline later. |
Each mode produces the same envelope: one User-agent group, the rules that come from your choice, and one sitemap URL derived from the origin you entered. If your sitemap lives somewhere else — for example on a separate domain or under a non-default path — edit that line in the downloaded file before you publish. The generator does not look up your sitemap for you, and intentionally so, because fetching it would mean sending your domain to a server.
Anatomy of a Generated robots.txt File
The output is short by design. Knowing what every line does makes it easier to review before publishing and easier to explain to anyone who later inherits the site. The table below walks through each part of a typical selective-mode result.
| Line in the file | Purpose | Where it comes from |
|---|---|---|
| User-agent: * | Names the wildcard product token so the rules apply to any crawler that does not have its own more specific group. | Always emitted, per RFC 9309. |
| Disallow: /private/ | Asks compliant crawlers not to fetch URLs whose path begins with /private/. | One per line in selective mode; wildcard * and trailing $ are preserved verbatim. |
| Sitemap: https://example.com/sitemap.xml | Tells crawlers the absolute location of your sitemap. | Built from the origin you entered; if you typed a deeper URL, the path, query, and fragment are stripped first. |
Two minor details often surprise first-time users. First, hash signs are not allowed inside a rule, because a leading # would turn the rest of the line into a comment. Second, paths are compared case-sensitively from the very first character of the URL path: /Private and /private are different rules, and the longest match wins when two rules overlap, as specified in RFC 9309.
Generate a robots.txt File in Your Browser
The following steps describe the verified operating flow of the Robots.txt Generator. No external service is contacted at any point, and everything you enter lives in the current tab until you clear it.
- Enter the complete credential-free HTTP or HTTPS URL for the website origin. Type the full address, including the scheme and any explicit port you serve on, such as https://www.example.com or http://staging.example.com:8080. The tool normalizes this to its origin so the sitemap line it generates matches the exact scheme, host, and port you intend to publish under.
- Choose an overall crawler policy. Pick allow all, block all, or selective disallow. If you choose selective disallow, you will be invited to list the paths you want compliant crawlers to skip; the other two modes skip straight to generation.
- Add one disallowed path per line. Control characters and hash signs are rejected; every non-empty rule must start with a slash, and the list accepts up to 50 unique rules. Exact duplicates are removed in the order they were first entered, so retyping the same path will not double it up. You may include wildcard asterisks and a trailing $ to mark end-of-path, because those are part of the standardized matching language.
- Generate and inspect the text. Trigger generation and read the resulting file from top to bottom. Confirm the User-agent line is the wildcard, your Disallow paths are spelled exactly as you intended (including capitalization), and the Sitemap line points at the correct origin plus /sitemap.xml.
- Compare with the existing live file before publishing. If your site already serves a /robots.txt, fetch the current one and diff it against the new output. Preserve any crawler-specific groups you deliberately maintain, and keep a copy of the old file so you can roll back if a crawler behaves unexpectedly.
- Publish the file as /robots.txt at the top level of the matching origin. Use a lowercase filename, serve it as text/plain, and place it at the top level of the exact scheme, host, and port it governs. A file on a different subdomain, a subdirectory, or a different protocol will not govern the intended origin.
Publishing Checks Before You Replace the Live File
Generating the file is only half the work. Crawlers learn your new rules only after they refetch /robots.txt, and any error in placement is invisible until traffic spikes or rankings shift. Run through this short checklist each time you replace the production file.
- Scheme, host, and port match. A staging file at http://staging.example.com:8080/robots.txt does not govern https://www.example.com/robots.txt, and vice versa. Confirm the URL you typed in the first step corresponds to where the file now lives.
- Sitemap line is still correct. If your sitemap moved since the last file was published, edit the generated line before uploading. If the file was generated from a deep URL, the tool already stripped it to the origin; check that the resulting path is the one you actually serve.
- Crawler-specific groups are preserved. The generator writes one general group by design. Any additional blocks you maintain for a particular crawler — for instance, ask a specific news crawler to ignore a section — need to be added back from your notes, otherwise the new file will silently remove them.
- Important URLs behave as expected. Use a robots.txt tester from the relevant search engine where available. A blocked URL should be reported as blocked, and an allowed URL should fetch normally. If the tester disagrees with your file, recheck capitalization and trailing slashes.
- Old file is archived. Keep the previous /robots.txt on your local machine so you can restore it quickly if a rollback is needed.
Path-Matching Cases That Catch People Out
Because every rule begins with a slash and uses standardized matching, a few patterns routinely produce wrong results when copied from older guides or from memory. The list below is not exhaustive, but it covers the cases that show up most often when a free robots.txt generator is used.
- Case matters. Disallow: /draft does not block /Draft or /DRAFT, because per RFC 9309 the longest match is taken with a case-sensitive comparison. If you want both, list both, or widen the rule.
- Trailing slash versus no trailing slash. Disallow: /draft matches every URL whose path starts with /draft, including /drafts; Disallow: /draft$ matches only the exact /draft path. Use the dollar anchor when you want one URL and not a prefix.
- Empty paths are not allowed. Each rule must start with /. A rule like Disallow: with nothing after it is ignored.
- Comments inside rules break the line. Anything that begins with a hash is treated as a comment, so write rules such as Disallow: /private/ without inline notes.
- Crawl-delay is not standardized. The generator does not emit Crawl-delay because it is outside RFC 9309 and not supported consistently. If you need throttling, look at server- or platform-specific settings rather than bolting it into the robots file.
None of these checks require a server round-trip — they all happen while you read the generated text in the browser. That is one of the quiet benefits of running a generator locally: each adjustment costs only a fresh keystroke, a new generation, and a quick read-through.
What "Free" and "Local" Actually Buy You
The Robots.txt Generator assembles the file locally in the current tab; it does not send configuration data to a remote server for processing. The trade-off is equally clear — the generator does not fetch your current /robots.txt, does not validate server responses, does not submit the file to any search engine, and does not confirm that a crawler has refreshed its cache. Those actions require access to the deployed site and to each engine's own tools.
If your site sits on a managed platform that hides the origin filesystem, you can still use the generator and paste the result into the platform's robots.txt editor — the output is plain UTF-8 text and works anywhere a robots file is accepted. A walk-through for one common platform is available in this local-output guide, and Google's own guidance on creating and placing the file is summarized at Google's Crawling Infrastructure documentation.
Finally, remember the protocol's own caveat: robots.txt is a public request to compliant crawlers, not a security boundary. Malicious clients ignore it, every path is visible to anyone who requests the file, and a URL blocked from crawling can still appear in search results if a search engine discovers it through links. Pair the generated file with whatever page-level indexing control, HTTP header, removal workflow, or authentication mechanism matches the actual outcome you need.