To get a robots.txt file, you either fetch the existing one hosted at a site's root or generate a fresh, standards-aligned file in your browser — both approaches return the same simple plain-text format defined in RFC 9309. A robots.txt file is a small plain-text document that lives at the top level of a website and tells compliant crawlers which paths they may or may not request. It follows the Robots Exclusion Protocol standardized in RFC 9309, which defines a group-and-rule model: each group starts with a User-agent line and contains zero or more Allow or Disallow rules. Because the format is plain UTF-8 text, anyone can read it directly in a browser, and anyone with write access to a site's root can publish one. The file is always served as text/plain, must be named robots.txt in lowercase, and lives at the exact scheme, host, and port it governs. This article covers both ways to get one — retrieving an existing file from a live site and generating a new file locally using the Robots.txt Generator — so you can choose the method that fits your goal, whether that is auditing another site or building one for your own.
Two ways to get a robots.txt file
The phrase "get a robots.txt file" can mean two different things depending on your situation, and the right method depends on which one you need.
Retrieve an existing file. Every compliant website serves its robots.txt at the root of its origin. If you want to inspect what a site is currently telling crawlers, append /robots.txt to the site's origin URL and load it in a browser or with an HTTP client. For example, requesting https://example.com/robots.txt returns the live file. This is the fastest way to get a robots.txt file when you want to audit a competitor, debug crawling behavior, or confirm that a deployment is live. The file is always public, so this approach requires no authentication.
Generate a new file. If you are setting up a site for the first time, replacing a missing file, or rebuilding rules after a redesign, you need to create the text and publish it yourself. Doing this in your browser with the Robots.txt Generator means the URL, paths, and resulting text never leave your tab — the generator normalizes the origin, validates each path against the RFC 9309 syntax, and emits plain UTF-8 text you can download. You then place that text at the origin's root as /robots.txt.
The two methods are complementary. Retrieval tells you what is live today; generation gives you a standards-aligned starting point you can compare against the current file before publishing.
How to generate a robots.txt file with the Robots.txt Generator
The generator follows the RFC 9309 group-and-rule model and writes a single wildcard User-agent group with either a global Allow or Disallow, or one Disallow line per path you enter. Follow these steps to get a fresh file:
- Enter the complete HTTP or HTTPS URL for the website origin. Use the scheme and host that govern the site you intend to publish to — for example, https://example.com. The generator reduces the URL to its origin so a page path, query, or fragment does not leak into the default sitemap line. Explicit ports are preserved.
- Choose an overall crawler policy. Pick one of three modes: Allow all (compliant crawlers may access every path), Selective (crawlers should avoid only the paths you list), or Block all (crawlers should avoid the entire origin). The choice controls the single rule the generator writes for the wildcard group.
- In Selective mode, enter one disallowed path per line. Every non-empty rule must start with a slash. The generator removes exact duplicates in first-seen order, caps the list at 50 rules, preserves wildcard asterisks and trailing dollar-sign anchors, and rejects hash signs so an unintended comment does not slip into the output.
- Generate and inspect the text. The output contains a User-agent: * group followed by either Allow: /, Disallow: /, or the Disallow lines you entered, plus a Sitemap: line pointing to <origin>/sitemap.xml. Read the text top to bottom and confirm that capitalization, prefixes, wildcards, and anchors match what you intended.
- Compare against the current production file before publishing. Download the generated text, open the live /robots.txt in another tab, and diff the two. Preserve any intentional crawler-specific groups from the old file and keep a copy of it for rollback. Publish the new text only as /robots.txt at the exact scheme, host, and port it governs, served as text/plain.
If your sitemap lives somewhere other than /sitemap.xml, edit the Sitemap line before publishing — the generator only writes the default location derived from the origin.
The three policy modes at a glance
Each mode produces a different wildcard group. The table below summarizes what the generator writes and when each mode fits a real goal. The exact rule text comes from the tool itself; the use-case descriptions are qualitative.
| Mode | Rule the generator writes | When it fits | Crawler behavior |
|---|---|---|---|
| Allow all | Allow: / | Public site with no restricted areas; you want full crawling and indexing | Compliant crawlers may request any path on the origin |
| Selective | One Disallow: line per entered path | Most production sites; you want specific folders (internal search, admin, staging) off-limits | Listed paths are requested to be avoided; everything else crawlable |
| Block all | Disallow: / | Staging environment, private portal, or pre-launch site not ready for indexing | Compliant crawlers asked to avoid the entire origin |
Selective mode is the most common starting point because most sites need a small handful of paths hidden from crawlers while leaving the rest of the origin open. The generator's 50-rule cap is a deliberate scope choice: anything beyond that usually signals the rules belong in a different control surface such as authentication, HTTP headers, or a search-engine removal tool.
RFC 9309 path-matching rules that affect your file
Because the generator follows the standardized protocol described in RFC 9309, the rules you enter are matched by compliant crawlers using the matching rules the protocol defines. Reviewing these before you publish prevents the most common surprises.
- Case-sensitive paths. RFC 9309 specifies that crawlers should compare paths case-sensitively, so /Private and /private are different paths. A Disallow rule for one does not cover the other.
- Longest matching rule wins. When more than one rule could match a URL path, the most specific (longest) match is used. A short Disallow: /draft does not express the same intent as Disallow: /draft$; the dollar sign anchors the rule to the exact path.
- Wildcards and end anchors are preserved. The protocol defines * as a wildcard and $ as an end-of-path anchor. The generator keeps both because they change which paths a rule actually covers.
- Matching starts at the path's first octet. Every valid Allow or Disallow pattern begins with a slash. That is why the generator rejects paths that do not start with / and why the default Allow-all rule is written as Allow: / rather than Allow: *.
These properties are also why a rule like Disallow: /draft does more than block a single page — it also blocks /drafts, /draft/2024, and anything else that begins with that prefix, unless a longer Allow rule overrides it.
Where and how to publish the generated file
A robots.txt file only governs the exact origin it is served from, so the location and serving details matter as much as the contents. The generator outputs the text, but it cannot place it on your server for you.
Publish the file at the top level of the origin's scheme, host, and port. For https://example.com that means https://example.com/robots.txt, not https://www.example.com/robots.txt, https://example.com/blog/robots.txt, or http://example.com/robots.txt. A file on a different scheme, host, port, or subdirectory does not govern the intended origin. Use a lowercase filename and serve it as text/plain so the response carries the correct MIME type.
If you maintain a separate sitemap at a non-default location, edit the generated Sitemap line before publishing. The generator appends Sitemap: <origin>/sitemap.xml by reducing the entered URL to its origin and appending the conventional filename. Resources such as the Google guide to creating a robots.txt file describe the same serving requirements from a search-engine perspective.
What robots.txt cannot do
Robots.txt is a public request to compliant crawlers, not a security boundary. Knowing what it does not do prevents misuse that creates real exposure.
- It is not access control. RFC 9309 explicitly states these rules are not authorization. A malicious client can ignore the file entirely, and every listed path is publicly visible because the file itself must be fetchable.
- It does not guarantee removal from search results. Blocking crawling does not stop a search engine from discovering a URL through links and retaining limited information about it without fetching the blocked page. To actually remove indexed content, use page-level indexing controls, HTTP headers, or the search engine's removal workflow.
- It is not encryption or privacy. Sensitive paths listed in robots.txt are not hidden from view — the file is open to anyone who loads the URL. Server-side authentication, authorization, and access logging are the appropriate controls for private content.
- It does not submit itself to search engines. The generator does not fetch your live file, validate server responses, submit the file, or confirm that a crawler has refreshed its cache. Those steps require access to the deployed site and search-engine tooling.
Treat robots.txt as a polite, public note to cooperating crawlers — useful for shaping what gets fetched, never a substitute for server-side controls.
Before replacing a production file
If a /robots.txt already governs a live site, a careless overwrite can quietly block important pages or expose internal paths you intended to keep out of indexing. Run through this short checklist before publishing the generated text:
- Download the generated file and the current production file, then diff them line by line.
- Preserve any intentional crawler-specific groups from the old file. The generator writes one wildcard group; if the old file addressed named crawlers with their own rules, decide whether to keep them alongside the new wildcard group.
- Keep a copy of the old file for instant rollback.
- Test important public and blocked URLs against the relevant search-engine tester where one is available.
- Confirm the file is reachable at the exact scheme, host, and port you intend, served as text/plain.
When you have done all of that and the diff looks correct, replace the file. The generator has done its part: a standards-aligned, locally produced plain-text file with the right group structure and a sitemap pointer — ready for review and publication. For a deeper walk-through of how retrieval and generation fit together for your own site, the practical guide on getting a robots.txt file for your website extends the workflow with deployment specifics.