The best robots.txt file for WordPress is a short, standards-aligned plain-text document at your site root that uses one User-agent: * group, blocks crawler access only to paths you intentionally want hidden from compliant bots, leaves all other content crawlable, and ends with a single Sitemap line. It follows the rule model from RFC 9309, the Robots Exclusion Protocol, so each Disallow rule starts with a forward slash, case is matched exactly, and crawlers pick the longest matching rule when several apply. A WordPress-specific file typically includes Disallow lines for internal search results, feed endpoints, and any custom draft or staging folders you do not want indexed, while leaving posts, pages, categories, and the public-facing media library open. Generated locally in the browser, the file is served as text/plain and must live at the top level of the exact scheme, host, and port it governs, not on a subdomain or under a subdirectory. Used this way, robots.txt keeps low-value URLs out of the crawl queue without locking real content behind a blanket block.
Why a Standards-Aligned robots.txt Matters for WordPress SEO
WordPress sites publish a lot of URLs by default: posts, pages, category archives, tag archives, author archives, attachment pages, internal search results, RSS feeds, trackbacks, embeds, REST API endpoints, and admin AJAX calls. Every one of those endpoints is theoretically reachable by a compliant crawler. When search engines spend crawl budget on internal search results, feed URLs, or administrative endpoints, they have less time for the posts and pages that actually drive traffic. A carefully scoped robots.txt file redirects that budget toward the URLs you want indexed while politely asking crawlers to skip the rest.
Standards alignment matters because most crawlers, including Google's, follow RFC 9309 rather than older or platform-specific dialects. Sticking to the standardized group-and-rule model means your file behaves predictably across engines and is less likely to break after a WordPress core update, a plugin change, or a server migration. A file that uses non-standard shortcuts or duplicate groups can produce inconsistent behavior, where one crawler honors your intent and another ignores the file altogether.
Finally, WordPress sites are often hosted on subdomains, behind reverse proxies, or under HTTPS-only redirects. A file served from the wrong scheme, host, or port does not govern the origin you intend. Building the file from your actual production origin, and verifying the final URL before publishing, closes that gap.
WordPress Paths Worth Disallowing (and a Few to Leave Alone)
A "best" robots.txt for WordPress is selective rather than maximal. The goal is to block URL patterns that do not deserve indexation, not to hide the site. The patterns below are commonly added to a WordPress robots.txt file; include only the ones that match how your site is actually structured.
- /?s= internal search result URLs. These create thin, near-duplicate pages that compete with real content.
- /search/ same idea, different URL shape.
- /feed/ and /feed/atom/ RSS and Atom feed endpoints. Useful for subscribers, rarely useful in search results.
- /trackback/ legacy trackback URLs that frequently produce spam.
- /wp-admin/ WordPress admin area. Block this in front of the public site; legitimate crawlers should not need it.
- /wp-includes/ core PHP and library files. Not intended as public content.
- /xmlrpc.php a common brute-force target; blocking it from compliant crawlers reduces noise.
- /readme.html, /wp-config-sample.php, /license.txt files that disclose version information.
Paths you should generally leave open include /wp-admin/admin-ajax.php, because many themes and plugins rely on this endpoint for front-end requests and blocking it breaks functionality, and /wp-content/uploads/, because your public media library is what search engines need for image search. Posts, pages, categories, and product archives are the indexable content of the site and should stay open. If a path is not on this list and you are unsure, leave it open. A blocked URL does not necessarily disappear from search results, and over-blocking one important section is harder to recover from than leaving a few low-value pages crawlable.
Generate the Best robots.txt File for WordPress
The Robots.txt Generator builds the file entirely in your browser. Nothing about your site or your rules leaves the current tab, which means you can iterate without a staging server or a plugin. Follow the steps below to produce a WordPress-ready file.
- Enter the full HTTP or HTTPS URL of your WordPress site origin. Use the scheme, host, and port your visitors actually use, for example https://example.com, and avoid adding page paths, query strings, or fragments, because the generator reduces the input to its origin to keep the sitemap line honest.
- Choose a crawler policy. Select "Allow everything" if you want compliant crawlers to reach every URL on your site, which produces a minimal allow-all file. Select "Block selected paths" if you want to list specific URL patterns to block. Select "Block everything" only if you intend to take the site out of indexing entirely.
- If you chose selective blocking, enter one disallowed path per line. Each non-empty line must begin with a forward slash. Add the WordPress paths from the previous section that match your site, for example /?s=, /search/, /feed/, /trackback/, /wp-admin/, /wp-includes/, and /xmlrpc.php. The tool accepts up to 50 unique rules, removes exact duplicates in first-seen order, and preserves wildcard asterisks and trailing dollar anchors because RFC 9309 defines those special matching characters.
- Click generate. The output is plain UTF-8 text with one wildcard user-agent group, your selected rules, and a single Sitemap line built from the normalized origin plus /sitemap.xml.
- Inspect the text. Confirm capitalization, prefixes, and anchors. Remember that /Private and /private can mean different things, and Disallow: /draft is not equivalent to Disallow: /draft$.
- Download the result, compare it against your current production file, and only then publish it as /robots.txt at the top level of the exact origin you entered. Serve it as text/plain and keep a copy of the old file for rollback.
| Mode | Output Behavior | Typical WordPress Use Case | Sitemap Line? |
|---|---|---|---|
| Allow everything | Emits User-agent: * and Allow: / | Default state for content-driven blogs and stores | Yes, origin + /sitemap.xml |
| Block selected paths | Emits User-agent: * followed by one Disallow: line per unique input | Production WordPress site hiding internal search, feeds, and admin | Yes, origin + /sitemap.xml |
| Block everything | Emits User-agent: * and Disallow: / | Temporary takedowns, staging environments, pre-launch locks | Yes, origin + /sitemap.xml |
Review and Publish Without Breaking the Live Site
A robots.txt file is small but high-impact. Replacing a working file with a broken one can deindex real content or strand crawlers, so treat the publish step like any other production change. Pull the live file before you change it and compare its current contents, including group order, any crawler-specific blocks for Googlebot or Bingbot, and any pre-existing sitemap lines, against what the generator produced. If you need help fetching the current file alongside the new one, the guide on how to retrieve or generate a robots.txt file walks through the retrieval step. Preserve any intentional crawler-specific groups you find, because selective blocking in the generator applies to the wildcard group only and does not impersonate per-engine behavior.
Test before pushing. If your search engine offers a robots.txt tester, run the new file through it and check representative URLs from each important template: a public post, a category archive, an internal search result, and the admin endpoint. Confirm that the expected URLs are marked allowed and the blocked ones are marked disallowed. Then publish the file as /robots.txt at the top level of the exact origin. A file on a subdirectory, another subdomain, or a different protocol does not govern the site you intended. Keep a rollback copy so you can restore the old file quickly if a crawler misbehaves or a critical URL is unexpectedly excluded.
When robots.txt Alone Is Not Enough
robots.txt is a public request to compliant crawlers, not an access control. A malicious client can ignore the file entirely, and every path you list is publicly visible. If your goal is to keep a page private, use server-side authentication and authorization: WordPress nonces, password protection, role-based access, or a real authentication layer in front of the application.
Blocking crawling also does not guarantee that a URL disappears from search results. Search engines can discover a URL through inbound links and retain limited information about it without ever fetching the page. If a URL is already indexed and you need it gone, use the search engine's removal workflow, page-level indexing controls such as noindex, or HTTP headers like X-Robots-Tag. Reserve robots.txt for the URLs you do not want crawled in the first place.
Finally, do not stack directives you do not fully understand. The generator intentionally omits Crawl-delay because the directive sits outside RFC 9309 and is not supported consistently across crawlers. Combining non-standard directives across plugins, themes, and a manual file tends to produce rules that cancel each other out, which is the opposite of a best robots.txt file for WordPress.