A robots.txt file is a plain UTF-8 text file placed at the top level of a website origin that tells compliant crawlers which paths they may request and which they should avoid. It uses one or more user-agent groups followed by Allow, Disallow, and Sitemap rules standardized by RFC 9309, and it works entirely through cooperation rather than enforcement: a compliant crawler honors the request, a misbehaving one can ignore it. The file lives at exactly /robots.txt on the same scheme, host, and port it governs, and a file on a subdomain or under a directory has no authority over the rest of the site. Because the rules are public and are not a security control, robots.txt is a crawl-management tool, not an access-control tool, and it cannot hide a page from anyone who knows or guesses the URL. A small generator that runs in the browser produces a working file with the right structure, a sitemap reference, and a copy you can review before deploying.
What a robots.txt file does for your website
The file has a single job: communicate crawl preferences to automated clients that follow the Robots Exclusion Protocol. When a compliant crawler prepares to fetch a URL, it first requests /robots.txt from the site's origin and reads the rules before requesting any page. Three rule keywords are standardized: Allow to permit a path, Disallow to ask a crawler to skip a path, and Sitemap to advertise the location of an XML sitemap. Everything else, including Crawl-delay, is outside the standard and is interpreted inconsistently or not at all across crawlers.
A typical file looks like this: a User-agent line identifies which crawler the following rules apply to, the wildcard asterisk matches any crawler that has no more specific group, and one or more Allow or Disallow lines express the path preferences. The Sitemap line at the end gives crawlers a direct pointer to the canonical URL list, which speeds up discovery without depending on external sitemaps or inbound links.
Because the file is fetched before any page load, a misconfigured robots.txt — a stray Disallow: / in the wrong place — can accidentally hide the entire site from search engines long before anyone notices the traffic drop. That is why a generator that produces a small, predictable file is more useful than a copy-paste snippet from an old blog post.
Why a browser-only generator is the safer approach
A browser-only generator handles the file locally, so the site origin, the path list, and the resulting text never leave the current tab. That matters for two reasons. First, no third-party server ever sees your URL structure, which removes any concern about logs, telemetry, or accidental retention of internal paths. Second, because the file is produced by JavaScript on the page rather than fetched from a remote API, you can keep working even if your network blocks third-party hosts or you are working on a staging environment behind a firewall.
For most websites, a standards-aligned generator is enough. The Robots.txt Generator takes a single HTTP or HTTPS origin, lets you pick one of three overall crawler policies, and emits a clean UTF-8 text file with a sitemap line derived from the origin. It deliberately writes one wildcard user-agent group instead of trying to mimic the proprietary directives that individual search engines have layered on top of the standard over the years. That focused output is easier to read, easier to audit, and easier to publish.
Generate a robots.txt file for your website
- Open the Robots.txt Generator and paste the complete HTTP or HTTPS origin of your site, including scheme and host, into the URL field. Do not add a page path, query, or fragment; only the scheme, host, and optional port matter.
- Choose an overall crawler policy. Allow all writes Allow: /, Block all writes Disallow: /, and Selective mode reveals a text box where you enter one disallowed path per line.
- If you picked Selective, list each path you want to block on its own line. Every non-empty line must start with a slash, exact duplicates are removed in first-seen order, and the list is limited to 50 unique paths.
- Click Generate to produce the file text. The output includes a single User-agent: * group with your chosen rules and a Sitemap line built from the normalized origin plus /sitemap.xml.
- Download or copy the text, then compare it character by character with your current /robots.txt if one already exists. Preserve any intentional crawler-specific groups you have written manually before replacing the live file.
- Publish the result as /robots.txt at the top level of the exact origin you entered, serve it as text/plain, and keep a copy of the old file so you can roll back if something goes wrong.
What the generated file actually contains
The file produced by the generator is small on purpose. It contains a User-agent line that uses the asterisk wildcard, then either one Allow line, one Disallow line, or one Disallow line per path you entered, followed by a single Sitemap line that points at the origin plus /sitemap.xml. An allow-all file for https://example.com looks like this in plain text:
User-agent: *Allow: /Sitemap: https://example.com/sitemap.xml
A selective file with /admin and /private blocked reads as:
User-agent: *Disallow: /adminDisallow: /privateSitemap: https://example.com/sitemap.xml
A block-all file reads as:
User-agent: *Disallow: /Sitemap: https://example.com/sitemap.xml
If your real sitemap lives at a different path — for example /feeds/sitemap.xml or /sitemap_index.xml — edit the Sitemap line in the downloaded text before publishing. The generator normalizes the origin only, so it cannot guess an alternate sitemap location on its own. A URL containing a page path, query, or fragment is reduced to its origin so the default sitemap does not accidentally inherit an unrelated route, and explicit ports are preserved.
Rules that affect which paths get blocked
The standard that defines this file, RFC 9309, is short and very specific. Three behaviours govern how compliant crawlers interpret the rules:
| Behaviour | Source | Practical effect |
|---|---|---|
| Paths match from the first character of the URL path | RFC 9309 §2.2.2 | A rule of /admin matches /admin, /admin/login, and /administrator |
| Matching is case-sensitive | RFC 9309 §2.2.2 | /Private and /private are different paths |
| The most specific match wins | RFC 9309 §2.2.3 | A longer Disallow beats a shorter one for the same crawler |
Two special characters are part of the protocol and should be preserved when they appear in your path list. The asterisk is a wildcard that matches any sequence of characters, and a trailing dollar sign anchors the pattern to the end of the URL. Disallow: /draft$ therefore matches only /draft and not /drafts, while Disallow: /draft matches both. The generator preserves both characters but rejects a hash sign, because RFC 9309 treats anything from a hash to the end of the line as a comment; accepting a hash in your input would silently truncate the rule you meant to write.
Because rule matching begins at the start of the URL path, every non-empty Allow or Disallow pattern must begin with a slash. Empty rules are treated as no-op and are not written to the file. Review capitalization, prefixes, wildcards, and anchors before publishing, because a single extra character can change what the crawler actually fetches.
Where to publish the file so crawlers actually see it
Where you publish the file matters as much as what is in it. A compliant crawler fetches /robots.txt from the exact scheme, host, and port that owns the URL it is about to crawl, and ignores any other file it finds. The file lives at https://example.com/robots.txt, not at https://www.example.com/robots.txt if your site redirects from one to the other, and not at https://cdn.example.com/robots.txt if your pages are served from a different host. If you serve pages from both example.com and www.example.com and they are separate origins, each origin needs its own robots.txt.
The filename is case-sensitive on most servers, so use a lowercase /robots.txt, not /Robots.Txt or /ROBOTS.TXT. Serve it as text/plain so crawlers do not have to guess the content type. A common deployment mistake is to upload the file into a /public, /static, or /uploads directory; that location will not be discovered because the crawler asks for the top-level path.
Before replacing a production file, download the generated text, compare it line by line with the current live file, and keep a backup copy of the old file in case the new version accidentally blocks an important path. If your existing file contains crawler-specific groups, for example User-agent: Googlebot, preserve those lines and append the new wildcard rules below them rather than overwriting them.
Limits of robots.txt and what it cannot do
The most important limit is also the easiest to forget: robots.txt is a public crawler request, not access control. RFC 9309 says so directly. Private content still needs server-side authentication and authorization; a Disallow line on /internal will not stop a malicious client from requesting the URL, and it does not even stop a compliant crawler from discovering the URL through inbound links. Blocking crawling also does not guarantee that a URL disappears from search results — it may retain a snippet derived from external signals or earlier crawls. If the goal is to keep a page out of the index, use a robots meta tag with noindex on the page itself, an X-Robots-Tag header, an authentication layer, or the search engine's removal workflow.
The generator does not add Crawl-delay because the directive is not part of RFC 9309 and is not honored uniformly across crawlers. It does not fetch the existing live file, validate the server response after publishing, submit the file to any search engine, or confirm that a crawler has refreshed its cached copy. Those steps require access to the deployed site and the search engine's own tools. Use the generator to produce a clean, reviewable file, then perform those confirmations through the appropriate channels.
| Policy mode | Output User-agent group | Typical use |
|---|---|---|
| Allow all | Allow: / | Open sites that want every compliant crawler to read everything |
| Selective | One Disallow line per entered path | Sites that need to hide specific sections such as /admin or /search |
| Block all | Disallow: / | Staging environments, archives, or sites that are not ready to be crawled |
Do not combine directives from different tools without understanding how each crawler handles them. The generator writes one wildcard group on purpose so the output is auditable line by line, and any extra crawler-specific group you need can be added by hand before publishing.