A robots.txt file is a plain text file placed at the root of a website that tells cooperating web crawlers which paths they are allowed or not allowed to request. It is the very first file search engines look for when they visit a site, and it uses two core directives — Allow and Disallow — together with an optional Sitemap line to steer how bots traverse your content. Although the rules are voluntary and well-behaved crawlers honor them, the file remains the standard way to declare crawl preferences at the site level rather than the page level.
If you have ever searched for the best way to publish those rules, the answer is shorter than you might think: write the directives in a plain text editor, save the result as robots.txt, upload it to the document root of your domain, and verify it by loading the URL in a browser. The challenge is not the file itself but the discipline of getting the syntax right, keeping the path list consistent with what your site actually serves, and avoiding the classic mistake of blocking content you wanted indexed. That is where a focused generator helps, because it formats the directives, escapes edge cases, and lets you inspect the output before anything reaches a server.
What robots.txt actually does
The Robots Exclusion Protocol, originally proposed in 1994 and still maintained as an informal community standard, defines how a site can advertise crawl rules to user agents such as Googlebot, Bingbot, and countless smaller crawlers. The standard is documented on the W3C community pages and on the maintainer's site, which describes the grammar for User-agent, Allow, Disallow, Sitemap, and a small set of modern extensions. Because the protocol is voluntary, a malicious bot can ignore it — that is one reason authentication, server-side access controls, and the noindex meta tag exist alongside it.
In practice, robots.txt serves three jobs. It keeps low-value paths (search result pages, internal search URLs, staging directories, temporary campaign folders) out of the crawl queue. It points crawlers to your XML sitemap so they can find the canonical version of each page. And it acts as a safety net when a sensitive area has accidentally been linked from elsewhere — you can block the path at the root while you decide on a stronger fix.
The core directives in plain English
Every robots.txt file is a sequence of one or more groups, and each group has the same shape: one or more User-agent lines naming the crawler the rules apply to, followed by the rules themselves. Rules are written one per line, starting with Allow or Disallow, then a path prefix that the matcher compares against the beginning of each requested URL. A Disallow: / line blocks the entire site for that user agent; an empty Disallow allows everything.
| Directive | What it means | Example |
|---|---|---|
User-agent | Names which crawler the following rules apply to. * matches all cooperating crawlers. | User-agent: * |
Disallow | Path prefix the crawler must not request. Empty value allows everything. | Disallow: /private/ |
Allow | Path prefix the crawler may request, used to carve exceptions inside a broader disallow. | Allow: /private/public-notes/ |
Sitemap | Absolute URL of an XML sitemap. Must be a full URL, not a relative path. | Sitemap: https://example.com/sitemap.xml |
Paths are matched as prefixes by default, so Disallow: /admin blocks /admin, /admin/login, and /admin-archive alike. Add a trailing slash when you mean a directory. Crawlers are case-sensitive, so Disallow: /Admin/ does not block /admin/, and that mismatch is one of the most common reasons a "blocked" path is still being crawled.
How to create robots.txt using a browser-based generator
A generator saves you from typos and case mistakes by handling the formatting, while keeping the path list under your control. The Robots.txt Generator on this site runs entirely in your browser and never sends your rules to a remote server, which matters when the file describes private sections of a production site. Here is the full workflow, step by step.
- Open the Robots.txt Generator in your browser. The tool loads with empty fields and no required login.
- Enter the complete HTTP or HTTPS URL for the website origin — for example
https://www.example.com. This value is reused to build an absoluteSitemapURL later. - Choose an overall crawler policy. Use "Allow everything" if you want a permissive baseline that only declares a sitemap; use "Disallow everything" for a temporary lockdown; or use "Selective mode" if you want to block a curated list of paths.
- If you chose selective mode, enter one disallowed path per line. Use directory slashes where appropriate (for example
/internal-search/) and keep casing consistent with what your server actually serves. - Click Generate. The tool renders the full file in a readable text area, including any default
User-agent: *group and aSitemapline built from your origin URL. - Inspect the output character by character. Confirm that each blocked path uses the casing your site uses, that the
Disallowlines are not so broad they cover URLs you want indexed, and that the sitemap URL resolves. - Compare the new text with the current production file before publishing. Open
https://your-site/robots.txtin another tab and diff the two files so you can spot accidental changes to existing rules. - Save the file as
robots.txtin plain UTF-8 text (no BOM, no rich formatting) and upload it to the document root of your domain so it resolves at/robots.txt.
Because the generator never transmits your configuration data, you can safely paste internal path patterns during drafting. The only data leaving your browser is whatever you choose to download or copy afterward.
Publishing and verifying the file
Once the file is saved and uploaded, verify it the way crawlers will: request the canonical URL directly in a browser, in curl, or in the testing tool inside your preferred search engine. The file must be served with a 200 OK status, as a text/plain (or compatible) response, and must not be wrapped in HTML, redirected, or blocked by a firewall rule. A non-200 response is treated by most crawlers as if no robots.txt existed, which means the default behavior is "everything allowed" — usually not what you want.
After publishing, give the major crawlers a chance to refresh their cached copy. Most engines re-fetch robots.txt within a day for active sites, but a manual fetch through the search console's robots.txt tester speeds up the cycle and shows you exactly which lines each crawler parsed. If you are rolling out a new block list on a large site, consider staging the changes behind a test origin first so you can confirm the rules behave as expected before they reach production.
Common pitfalls and how to avoid them
The same handful of mistakes account for most broken robots.txt files. Treating Disallow as authentication is the biggest category: the directive only asks cooperating crawlers not to fetch a path, so it does not hide a page from a user who knows the URL. For genuinely sensitive material, gate the path at the application layer or return a proper 401/403 response.
Other pitfalls worth checking for: writing the file in a rich-text editor that adds a BOM or smart quotes, using relative paths for the Sitemap directive, blocking the path that contains the sitemap itself, and confusing Disallow with the page-level noindex meta tag. Search engines document this distinction clearly: Disallow prevents crawling, while noindex prevents indexing, and the two are not interchangeable. If you want a page out of the index but still need crawlers to see it (for example to discover its canonical link), use noindex, not a blanket block.
| Pitfall | What goes wrong | Safer approach |
|---|---|---|
Saving as robots.txt.txt or inside a subfolder | Crawlers never find the file at /robots.txt | Upload to the exact document root, lowercase filename, no extension |
| Using a rich-text editor | Smart quotes or BOM break directive parsing | Use a plain text editor and save as UTF-8 without BOM |
| Casing mismatch | Disallow: /Admin/ does not block /admin/ | Match the exact casing the server uses for each path |
| Blocking the sitemap path | Crawlers cannot read the sitemap you are pointing them to | Keep Sitemap lines outside any blocked group, and verify the URL resolves |
Using Disallow for security | The directive is voluntary and easily ignored | Enforce access with server rules, auth, or a proper noindex tag |
Where robots.txt fits in a wider SEO setup
Robots.txt is one of several crawl-control levers, and it works best when paired with the others. A canonical tag tells crawlers which version of a duplicate page to index, an XML sitemap lists the URLs you actually want indexed, and a hreflang set tells multilingual crawlers which page belongs to which audience. You can draft each of these in the browser as well: the Canonical Tag Generator produces a clean rel="canonical" link element, the XML Sitemap Generator turns a reviewed URL list into a sitemap file, and the Hreflang Generator builds a consistent set of alternate links from locale-and-URL rows. Used together, these files give crawlers an unambiguous map of your site and reduce the chance that ambiguous signals end up in the index.
Before you publish a new robots.txt, run one final sanity check: open the file in a fresh browser window using its full production URL, copy the result, and paste it into a diff tool against the version you are about to deploy. A two-line diff catches more mistakes than a careful re-read, and it makes the change auditable for anyone else who works on the site later.
Related guide: How to Generate a QR Code: A Complete Guide | Lizely.
If you're weighing options, How Does Schema Markup Work: A Practical JSON-LD Guide covers this in detail.
If you're weighing options, Create an XML Sitemap in WordPress Without Plugins covers this in detail.