A robots.txt file is a small, plain-text file at the top of a website's origin that tells RFC 9309–compliant crawlers which paths they may request. It is not access control, encryption, or a removal tool — it is a polite, public request that cooperating crawlers honor while malicious clients can ignore. Creating one means choosing a crawler policy, writing a single User-agent group with matching rules, and publishing the result as /robots.txt at the exact scheme, host, and port it governs. The standard file is short enough to write by hand, but most teams prefer a generator that enforces the protocol's required characters, normalizes the origin, and prevents accidental comments or malformed paths. The Robots.txt Generator runs entirely in the browser, takes a site URL plus a policy choice, and produces a UTF-8 text file with a sitemap line so you can review it before uploading.
For a site that has never had a robots.txt file, the default behavior is "everything is crawlable" — crawlers simply treat the absence of the file as an open invitation. Adding the file does not block anything by itself; it documents the policy you actually want. For a site that already has a file, every change replaces the previous rules, which is why the publish and verify steps matter as much as the generation step.
What a Robots.txt File Actually Does
A robots.txt file is the standardized mechanism defined in RFC 9309 for site owners to publish crawler preferences. The current Robots Exclusion Protocol standardizes exactly two directive types: User-agent, followed by one or more Allow or Disallow rules. Per the IETF specification, a file begins with one or more groups; each group lists the products it applies to and the paths they should treat as off-limits or permitted.
When a compliant crawler visits your origin, it reads /robots.txt, picks the most specific matching group, and uses the longest-matching rule for any path. This is why a small file can govern an entire site: the wildcard token * matches any product that has no more specific group, the special character * inside a path matches any sequence of characters, and a trailing $ anchors the pattern to the end of the path. Two practical consequences follow directly from this matching model:
- Paths are compared case-sensitively, so /Private and /private are different patterns.
- A rule without a trailing $ matches everything that begins with the listed prefix, so Disallow: /draft blocks /draft, /drafts, and /draft/2024/01.
The file is requested over plain HTTP or HTTPS, served as text/plain, and must sit at the origin it governs. A robots.txt at https://www.example.com/blog/robots.txt does not govern https://www.example.com/, and a file on a different scheme or port does not apply to the alternate origin. The Robots.txt Generator follows this rule by normalizing every input URL to its origin before emitting the Sitemap line.
The Three Policy Modes Explained
Most sites need exactly one of three policies, and a good generator makes the choice explicit before producing any text. The Robots.txt Generator exposes the same three, and the table below maps each mode to its typical use case and the rule it emits.
| Policy mode | Typical use case | Output rule |
|---|---|---|
| Allow all | Public sites with no path restrictions | Allow: / |
| Selective disallow | Production sites hiding admin, search, cart, or staging paths | One Disallow line per submitted path |
| Block all | Previews, staging, and dev environments that must not be indexed | Disallow: / |
Each option produces one User-agent: * group, the matching rule, and a Sitemap line pointing at the origin's /sitemap.xml. Choosing allow all is not the same as doing nothing — without a file, crawlers still treat the site as fully open, but you lose the chance to point them at a sitemap and document intent. Choosing block all turns the site into a private preview; combined with HTTP authentication, it is the simplest way to keep a stage from appearing in search results. Selective disallow is the most common production choice because it keeps the public site crawlable while steering crawlers away from internal areas such as /admin, /search, /cart, or /preview.
Create a Robots.txt for a Website
Generate a standards-aligned robots.txt using the Robots.txt Generator in three short steps:
- Enter the complete HTTP or HTTPS URL for the website origin in the URL field. The tool reduces the URL to its origin (scheme, host, and port) so the Sitemap line points at the right place, even when you paste a deep page link.
- Choose an overall crawler policy. Pick allow all, block all, or selective disallow. For selective mode, enter one disallowed path per line — each path must start with a slash, you may submit up to 50 unique entries, and exact duplicates are removed in first-seen order.
- Generate and inspect the text, then publish it as /robots.txt only after comparing it with the current production file. The generator never fetches your live file, so a manual diff is what protects you from losing an intentional crawler-specific group.
Step two deserves a closer look because selective mode is where most mistakes happen. Wildcard asterisks and a trailing dollar sign are preserved because the protocol defines those special matching characters, so /private/* and /draft$ are valid patterns. Hash signs are rejected to avoid accidentally turning part of a user-entered rule into a comment, so anything like /admin # block admin area will fail validation. The list is capped at 50 rules to keep the file readable and to discourage using robots.txt as a privacy tool. All entered paths and generated text remain in the current tab — nothing is uploaded to a server.
For a deeper walkthrough of the same workflow with extra checks, see the practical guide to creating a robots.txt file.
Read the Generated Output Before Publishing
A correct robots.txt is short, so read it line by line. The generator always emits one User-agent: * group because RFC 9309 standardizes the wildcard as the fallback product, and the generator intentionally does not pretend to manage every search engine's private extensions. For a typical production site with two blocked paths, the output looks like this in structure:
User-agent: * Disallow: /admin Disallow: /search Sitemap: https://www.example.com/sitemap.xmlThe Sitemap line is built from your normalized origin, not the deep link you pasted. A URL containing a page path, query, or fragment is reduced to its origin so the default sitemap does not accidentally inherit an unrelated route. Explicit ports are preserved, so https://staging.example.com:8443 stays attached to its port. If your sitemap lives somewhere other than the origin's /sitemap.xml, edit the generated line before publishing.
Common Detail Errors
- Capitalization: /Admin and /admin are different patterns in RFC 9309, so write paths exactly as they appear in the URLs you want to block.
- Prefix behavior: Disallow: /draft blocks /draft, /drafts, and /draft/anything unless you anchor it with Disallow: /draft$.
- Wildcards: a path like /private/* matches every subpath under /private, which is usually what you want for an admin folder but worth confirming before publishing.
If you are inheriting an existing file, a single wildcard group will overwrite crawler-specific blocks. Download the new file, compare it with the current /robots.txt, preserve any intentional crawler-specific groups, and keep a copy of the old file for rollback. For a focused check of how 32 AI and AI-adjacent product tokens match your rules, see how to verify results from an AI bot robots.txt check.
Where and How to Publish /robots.txt
The file must be published as /robots.txt at the top level of the exact scheme, host, and port it governs, served as text/plain with a lowercase filename. A file on a subdirectory, another subdomain, or a different protocol does not govern the intended origin. The Robots.txt Generator outputs UTF-8 text that can be saved with any name, but only /robots.txt is the right deployment name.
For an Apache site, drop the file at the document root and confirm the MIME type is text/plain. For an Nginx static block, place it at the root path and check that the types directive includes text/plain. For a CMS, use the built-in robots editor or the file manager — both WordPress and Shopify have a documented route, and the generator's output is compatible with either. After uploading, fetch the file directly with curl or a browser at the exact origin URL and confirm the response body matches what you generated.
Deployment Pitfalls to Avoid
- Adding Crawl-delay, even though some search-engine documentation still mentions it. The generator omits Crawl-delay because it is not part of RFC 9309 and is not universally supported.
- Mixing directives from multiple sources without understanding how each crawler handles them. Combine only rules you can explain in plain language.
- Serving the file from a CDN with a cached compressed variant that the crawler cannot parse. The body is short, so caching rarely matters, but verify the live response.
Verify the Deployment and Test Key URLs
The generator does not fetch the existing live file, validate server responses, submit the file to a search engine, or confirm that a crawler has refreshed its cache. Those actions require access to the deployed site and search-engine tools. After publishing, perform four checks:
- Fetch the deployed /robots.txt with curl -I and curl to confirm HTTP 200, the text/plain content type, and the exact body.
- Test a public URL in the relevant search-engine URL tester to confirm the page is still eligible for crawling after the new file is in place.
- Test a blocked URL in the same tester to confirm the rule applies and the page is excluded from live results where intended.
- Submit the sitemap URL through the search engine's console if the platform supports it, and watch for a successful read in the next crawl cycle.
Per Google's crawling documentation, robots.txt rules take effect on the next fetch by each crawler, and search engines may retain limited information about a URL discovered through links even when crawling is blocked. That is why the verification step matters: a missing Allow or an over-broad Disallow becomes visible only after the crawler revisits the file.
What Robots.txt Cannot Do
Robots.txt is one control in a larger indexing system, and the limits of the standard are worth naming directly:
- It is not authentication or authorization. Compliant crawlers honor the rules, but a malicious client can ignore them, and every listed path is publicly visible. Never list a sensitive route as a substitute for login checks or server-side access control.
- It does not guarantee removal from search results. A search engine may discover a URL through links and retain limited information without fetching the blocked page. Use page-level indexing controls, HTTP headers, removal workflows, or authentication for the actual goal.
- It is not a privacy tool. The file itself is fetched by anyone, and paths inside it are not encrypted.
- It does not control every crawler. Only clients that choose to follow the protocol respect it; private crawlers and scrapers can ignore it.
The Robots.txt Generator is scoped on purpose. It does one thing: produce a short, RFC 9309–shaped file for one website origin, entirely in the browser, and it does not pretend to manage crawler-specific extensions, server configuration, or removal workflows. Once you understand the policy, the generator, and the publish step, you can swap a live file in minutes — and the next time you need to update the file, the same three-mode workflow applies.