Generating a robots.txt for a WordPress site means producing a small UTF-8 plain-text file at the exact root of your scheme, host, and port that lists crawler requests using the group-and-rule structure standardized in RFC 9309. A browser-based generator such as the Robots.txt Generator keeps every byte of your configuration inside the current tab — nothing is uploaded to a backend, fetched from the live site, or sent to a search engine. The file itself contains a single wildcard User-agent group, a chosen Allow or Disallow policy, and a sitemap line derived from the site origin you enter. Because the protocol is case-sensitive, longest-match, and starts comparing from the first octet of a URL path, every rule must start with a slash, exact duplicates collapse in first-seen order, and `/Private` and `/private` are not interchangeable. Those constraints, plus a deliberate 50-rule cap and a refusal to emit Crawl-delay, are what make the output honest rather than impressive on paper, and the rest of this article walks through how to use the generator on a WordPress workflow without depending on a plugin.
Why WordPress Sites Need a Standards-Aligned robots.txt
WordPress has long shipped a virtual robots.txt when none is uploaded, which means many installations never see a literal file on disk until they create one themselves. Plugins that emit sitemaps, internal search templates, feed URLs, and preview links all add to the URL surface a crawler can discover. Without a deliberate file, those URLs get crawled by default and a small but predictable set of admin, login, and theme paths sit one click away from any bot that respects the protocol.
A standards-aligned robots.txt gives you a single, version-controlled place to express what compliant crawlers should avoid. It does not authenticate anything, hide anything, or encrypt anything — that responsibility still belongs to the application — but it sets the polite boundary the major search engines read first. The Robots.txt Generator produces exactly one such file: a UTF-8 plain-text document with a wildcard User-agent group, an Allow or Disallow policy of your choice, and a sitemap line pointing at the origin you entered.
What a Generator Should and Should Not Do for WordPress
A trustworthy browser-side generator should respect the constraints of RFC 9309 without trying to imitate private crawler extensions. For a WordPress workflow that means:
- Normalizing the URL you paste into a credential-free origin so the sitemap line cannot accidentally inherit a path, query, or fragment from your browser tab.
- Writing a single wildcard group that applies when no more specific match exists, instead of stacking invented directives.
- Accepting only slash-prefixed rule patterns, preserving `*` and a trailing `$` as the protocol defines, and rejecting `#` so a stray character does not silently turn part of a rule into a comment.
- Removing exact duplicates in first-seen order and capping the list at 50 rules, so the result remains auditable on one screen.
- Refusing to add Crawl-delay because the standardized RFC 9309 rules do not include it and crawlers do not honor it consistently.
The same tool should also be honest about what it cannot do. It does not fetch the existing live file, does not submit the generated file to a search engine, does not validate the server response, and does not confirm that a crawler has refreshed its cache. Those actions require access to the deployed WordPress site and the search engine's own tooling, and pretending otherwise would put the configuration at risk of being silently corrupted by a background fetch.
Generate a Robots.txt File for WordPress
- Open the generator and paste your site origin. Enter the complete HTTP or HTTPS URL of the WordPress site you want the file to govern. Strip any path, query, or fragment before pasting so the tool reduces the value to a clean origin; explicit ports are preserved.
- Choose an overall crawler policy. Pick Allow all to write `Allow: /`, Block all to write `Disallow: /`, or Selective to keep the default allow behaviour and add explicit disallow lines.
- In selective mode, enter one disallowed path per line. Each non-empty rule must begin with a slash, exact duplicates collapse in first-seen order, and the list is capped at 50 entries. Wildcard asterisks and a trailing `$` are preserved as the special matching characters the protocol defines; hash signs are rejected.
- Generate and inspect the text. Read the output for one wildcard User-agent group, the Allow or Disallow rules you chose, and a `Sitemap:` line that points at the normalized origin plus `/sitemap.xml`.
- Compare with the current production file. Before publishing, download the generated text, open the live robots.txt, and diff them side by side. Preserve any intentional crawler-specific groups the generator did not write, and keep a copy of the old file for rollback.
- Publish the file at the exact root it governs. Upload the text as `/robots.txt` in lowercase on the same scheme, host, and port; serve it as `text/plain`. A file on a subdirectory, a different subdomain, or a different protocol does not govern your WordPress origin.
- Test the public and blocked URLs. Use the relevant search-engine tester to confirm that a few representative paths still resolve the way you intended, and re-check after any WordPress plugin update that touches the sitemap or rewrite rules.
Reviewing the Generated Output for a WordPress Site
Once the text is in front of you, four checks protect you from the most common WordPress mistakes. First, capitalization: RFC 9309 specifies that crawlers should compare paths case-sensitively, so `/Private` and `/private` are different rules and only one of them will match the URL the CMS actually serves. Second, anchors: `Disallow: /draft` and `Disallow: /draft$` do not express the same intent — the second one stops at the segment boundary while the first is a prefix match. Third, longest match: when several groups could apply, the most specific one wins, so the order you list rules matters less than their specificity. Fourth, the sitemap line: it is built from your normalized origin plus `/sitemap.xml`, and if your real sitemap lives elsewhere — for example behind a plugin that emits `/sitemap_index.xml` or a separate `/news-sitemap.xml` — edit that line before publishing.
| Policy mode | What the generator writes | Practical use on a WordPress site |
|---|---|---|
| Allow all | User-agent: *Allow: /Sitemap: …/sitemap.xml | Public blogs and content sites that want every URL discoverable. |
| Block all | User-agent: *Disallow: /Sitemap: …/sitemap.xml | Staging environments, pre-launch domains, or private builds. |
| Selective | User-agent: *Disallow: /first-pathDisallow: /second-pathSitemap: …/sitemap.xml | Production WordPress where admin, search, or internal facets must be off-limits while the rest stays crawlable. |
Replacing or Adding the File in WordPress
WordPress only honours a robots.txt that lives at the literal root of the origin you configured. If you upload the file to a subdirectory like `/wp-content/` or to a `www.` subdomain when your canonical origin is the apex, compliant crawlers will keep reading the virtual file WordPress generates instead. Two practical paths exist for putting your generated text in place: edit or upload a real file via SFTP, SSH, or your host's file manager; or, if your host blocks root writes, use a robots.txt plugin that writes the file on demand and verify that the result resolves at the bare origin.
Before you overwrite anything, compare the new text with the current production file the way the step-by-step section describes. Plugins sometimes add their own directives (a WooCommerce cart disallow, for example, or a multilingual switcher exclusion), and a generator that intentionally emits one general group will not reproduce those. Keep a copy of the old file, edit the generated `Sitemap:` line if your real sitemap sits at a different URL, and confirm the served Content-Type is `text/plain` with a lowercase filename. A small file like this is also the right place to revisit when you migrate WordPress to a new domain, switch from HTTP to HTTPS, or move from `www` to the apex, because the file is scoped to one specific scheme, host, and port.
Common WordPress robots.txt Patterns and Their Meaning
WordPress sites usually consider a small, predictable set of paths. The table below lists the patterns people most often paste into a generator and what they actually mean under RFC 9309; it does not compute a list for you, because every site's crawl budget and content strategy differ.
| Pattern | What it matches | Why WordPress sites consider it |
|---|---|---|
| /wp-admin/ | Every URL beginning with /wp-admin/ | The admin area is not meant for crawlers, and the trailing slash prevents a partial match with plugin directories. |
| /wp-includes/ | Every URL beginning with /wp-includes/ | Core PHP and JavaScript that rarely changes; crawling it wastes budget. |
| /?s= | Internal search queries via the s parameter | Search result pages are usually thin and create near-duplicate content. |
| /search/ | Pretty permalink search results | Same intent as above on sites with rewritten search URLs. |
| /author/ | Author archive pages | Useful on multi-author blogs to keep archive facets out of the index. |
| /feed/ | RSS and Atom feed URLs | Optional; some publishers keep feeds in the index, others block them. |
After publishing, remember that blocking crawling does not guarantee removal from search results. A search engine may still discover a URL through inbound links and keep a limited snippet without fetching the blocked page. Use the appropriate indexing or removal control — a page-level noindex meta tag, an HTTP header, or an authentication gate — when the goal is actually to keep a page out of results. The RFC 9309 specification is explicit that these rules are not access authorization, and the Google crawling infrastructure documentation describes how its crawler reads the file in practice. For a wider, environment-agnostic walkthrough of how the file works and how to audit an existing one, see the how to create a robots.txt file in WordPress practical guide, which complements the generator-driven flow above.
For a deeper look, see Generate llms.txt From a Storybook Docs Site.