Skip to content
Lizely

Robots.txt Generator

Create a standards-aligned robots.txt file for one website without sending configuration data to a server.

Privacy: your files never leave your device. All processing happens locally in your browser.

How to use

  1. 1.Enter the complete HTTP or HTTPS URL for the website origin.
  2. 2.Choose an overall crawler policy and, for selective mode, enter one disallowed path per line.
  3. 3.Generate and inspect the text, then publish it as /robots.txt only after comparing it with the current production file.

About Robots.txt Generator

Robots.txt Generator creates a small crawler instruction file for a website entirely in the browser. Enter an HTTP or HTTPS site URL, choose whether compliant crawlers may access everything, should avoid selected paths, or should avoid the entire site, and generate plain UTF-8 text. The output includes one wildcard user-agent group and a sitemap URL derived from the site origin.

The generator follows the group and rule model standardized by RFC 9309. A group begins with User-agent, followed by Allow or Disallow rules. The wildcard product token applies when a crawler has no more specific matching group. This focused tool intentionally generates one general group instead of pretending to manage every search engine’s private extensions.

Selective mode accepts one path pattern per line. Each non-empty rule must start with a slash, exact duplicates are removed in first-seen order, and the list is limited to 50 rules. Wildcard asterisks and a trailing dollar sign are preserved because the protocol defines those special matching characters. Hash signs are rejected to avoid accidentally turning part of a user-entered rule into a comment.

Rule matching begins at the start of a URL path. RFC 9309 specifies that crawlers should compare paths case-sensitively and use the most specific matching rule. Consequently, /Private and /private can mean different things, and Disallow: /draft does not necessarily express the same intent as Disallow: /draft$. Review capitalization, prefixes, wildcards, and anchors before publishing.

Allow all mode writes Allow: /. Block all mode writes Disallow: /. Selective mode writes each requested Disallow line. The tool then adds Sitemap: followed by the normalized origin and /sitemap.xml. A URL containing a page path, query, or fragment is reduced to its origin so the default sitemap does not accidentally inherit an unrelated route. Explicit ports are preserved.

The output must be published as /robots.txt at the top level of the exact scheme, host, and port it governs. Use a lowercase filename and serve it as text/plain. A file on a subdirectory, another subdomain, or a different protocol does not govern the intended origin. If your sitemap has another location, edit the generated line before publishing.

Robots.txt is not authentication, authorization, encryption, or a privacy control. RFC 9309 explicitly says these rules are not access authorization. Compliant crawlers may honor the requests, but a malicious client can ignore them, and every listed path is publicly visible. Never list a sensitive route as a substitute for login checks or server-side access control.

Blocking crawling also does not guarantee that a URL disappears from search results. A search engine may discover a URL through links and retain limited information without fetching the blocked page. Use appropriate page-level indexing controls, HTTP headers, removal workflows, or authentication according to the actual goal. Do not combine directives without understanding how each crawler handles them.

The generator does not add Crawl-delay because it is not part of RFC 9309 and is not universally supported. It does not fetch the existing live file, validate server responses, submit the file to a search engine, or confirm that a crawler has refreshed its cache. Those actions require access to the deployed site and search-engine tools.

Before replacing a production file, download the result, compare it with the current file, preserve any intentional crawler-specific groups, and test important public and blocked URLs with the relevant search-engine tester where available. Keep a copy of the old file for rollback. All entered paths and generated text remain in the current tab.

Methodology & sources

Normalize a credential-free HTTP(S) URL to its origin; require 1–50 unique slash-prefixed paths in selective mode; preserve RFC-defined wildcard and end-anchor characters; reject control characters and comments; emit one User-agent: * group with Allow: /, Disallow: /, or the selected Disallow paths; append an origin-relative sitemap URL; and produce UTF-8 plain text locally. Disclose longest-match, case sensitivity, scope, and the absence of access-control guarantees.

Frequently asked questions

Does robots.txt protect private pages?
No. It is a public crawler request, not access control; private content requires server-side authentication and authorization.
Why does every path begin with a slash?
Allow and Disallow patterns match URL paths from their first octet, and valid path patterns begin with /.
Does blocking a URL remove it from search results?
Not necessarily. Crawling and indexing are separate, so use the appropriate indexing or removal control for that goal.
Why is Crawl-delay omitted?
It is outside the standardized RFC 9309 rules and is not supported consistently across crawlers.

Related tools

SEO & Webmaster guides

View all