WordPress serves a virtual robots.txt containing only two lines — User-agent: * and Disallow: /wp-admin/ — whenever a physical /robots.txt file is absent from the site root. The text is generated by WordPress core on the fly and is not stored on disk, which is why you never see it inside your wp-content directory and why editing your theme will not change it. Most SEO guides treat this virtual output as the "default robots.txt for WordPress," but it is really a placeholder: it has no sitemap reference, no host directive, and no protection for staging areas, draft folders, plugin directories, or feed URLs that you may want to keep out of crawlers. For a site that wants deterministic control over how compliant crawlers behave, the recommended move is to publish your own physical /robots.txt at the root and treat that file as the real default. The Robots.txt Generator builds that file in your browser using RFC 9309 group and rule syntax, then appends a sitemap line derived from the URL you enter so you do not have to hand-edit the result. Everything happens locally — the URL you type, the paths you choose to disallow, and the file you download never leave the tab — which matters when you are working on a private staging environment or a client project you would rather not upload anywhere.
What WordPress Ships as a "Default" robots.txt
When WordPress is installed and no /robots.txt file is uploaded to the document root, WordPress intercepts requests for that path and serves its own virtual content. That virtual response is the same across every standard WordPress installation and consists of User-agent: * followed by Disallow: /wp-admin/. There is no reference to a sitemap, no crawl-delay, and no protection for /wp-includes/, /feed/, author archives, draft previews, or staging subdomains. SEO plugin suites such as Yoast and Rank Math sometimes append extra comments and rules inside this virtual response, but only when they are active and configured to do so; on a stock WordPress install with no SEO plugin, the output is the two-line minimal version.
The reason this matters for your default robots.txt plan is that the virtual output is invisible to your file manager. You cannot open it in an editor, version-control it, or diff it against a backup. As soon as you place a physical file at the root, WordPress stops generating the virtual response and serves your file instead — and that is the moment the file becomes genuinely yours.
Why a Physical File Beats the Virtual Default
A physical /robots.txt gives you four concrete advantages over the virtual placeholder that WordPress otherwise emits:
- Predictable content. The exact bytes you uploaded are the exact bytes a crawler sees. No theme, plugin, or core change can silently rewrite your rules.
- Survives updates. WordPress core updates, theme switches, and plugin reinstalls all leave a physical file alone, so your crawler policy does not regress.
- Can include a sitemap line. The standard robots.txt group and rule model allows a Sitemap: line, and most major crawlers read it. The virtual response does not include one, so you lose that discovery hint by default.
- Survives security plugins and caching layers. Some caching and firewall plugins will rewrite or strip a virtual response. A static file at the web server level is harder to mangle.
For these reasons, treating a hand-built physical /robots.txt as your "real" default — even if the first version only contains Allow: / plus a sitemap line — is the safer baseline for any WordPress project.
Picking the Right Mode for a WordPress Starter
The Robots.txt Generator offers three modes for the one wildcard group it emits, and each maps to a different WordPress starting point.
| Mode | Lines written under User-agent: * | Best WordPress use case |
|---|---|---|
| Allow all | Allow: / | Most production sites that want every public URL crawlable and only need a sitemap reference |
| Selective disallow | One Disallow: line per path you enter | Sites that want to keep /wp-admin/, draft folders, internal search, or /feed/ out of crawlers without hiding everything |
| Block all | Disallow: / | Staging environments, sites under maintenance, or dev mirrors that should not be indexed at all |
Allow all mode is the closest match to what most people mean by a default robots.txt for a WordPress site: open the door to compliant crawlers, publish a sitemap line, and move on. Selective mode is what you grow into once you know which paths are wasting crawl budget or leaking internal URLs. Block all mode is for cases where the entire site should be invisible to crawlers until you lift the restriction.
How to Create a Default robots.txt for WordPress
Follow these steps to produce a standards-aligned starter file for your WordPress site using the browser-based generator.
- Open the Robots.txt Generator in your browser. No data is sent to a server — the URL you enter stays in the current tab.
- Type the full origin of your site in the URL field, including the scheme. For a production WordPress site, that is normally https://yourdomain.com (with an explicit port if you use one, for example http://staging.yourdomain.com:8080). The generator normalizes the URL down to its origin so the sitemap line does not accidentally inherit a page path, query, or fragment.
- Choose Allow all for a minimal default that mirrors the typical WordPress starting point. Switch to Selective disallow only when you have a specific list of paths to hide, and switch to Block all for staging or maintenance windows.
- If you picked selective mode, enter one path per line, with each rule starting with /. Exact duplicates are removed in the order you entered them, the list is capped at 50 rules, hash signs are rejected, and wildcard asterisks plus a trailing dollar sign are preserved because RFC 9309 defines them as matching characters.
- Click Generate and read the resulting plain-text block. Confirm the single User-agent: * group, confirm the Allow: or Disallow: lines match what you intended, and confirm that the appended Sitemap: line points at <your-origin>/sitemap.xml. If your sitemap lives somewhere else, edit that line before saving.
- Download the file as robots.txt (lowercase, exactly). Open the current production file at https://yourdomain.com/robots.txt in another tab, diff the two side by side, and confirm that nothing important from the old file is missing. Keep a copy of the old file for rollback.
- Publish the new file as /robots.txt at the exact scheme, host, and port it governs, served as text/plain. A file in a subdirectory, on a different subdomain, or under a different scheme does not govern the origin you intended.
Step 6 is the one most teams skip. A backup and a diff take thirty seconds and protect you from accidentally removing a crawler-specific group or a path the previous owner added on purpose.
Where to Upload robots.txt in WordPress
There are four reliable ways to put the file on the server. Pick one and stick with it so the file does not end up in two places.
- FTP/SFTP or SSH. Connect to your hosting account and upload robots.txt to the document root — the same directory that contains wp-config.php, wp-admin/, and wp-includes/. This is the most direct method and works on every host.
- Hosting file manager. Most managed WordPress hosts (Kinsta, WP Engine, SiteGround) expose a file manager in their control panel. Upload the file to the document root the same way.
- cPanel or Plesk File Manager. Same target directory — the document root, not the wp-content folder inside it.
- WordPress plugin. SEO plugins such as Yoast and Rank Math include a robots.txt editor inside the dashboard. Using that editor writes the file for you through the WordPress filesystem API. For a deeper walkthrough of dashboard and plugin paths, see How to Create a Robots.txt File in WordPress.
After uploading, load https://yourdomain.com/robots.txt in a private browser window and confirm you are reading the new content. If you still see the two-line virtual output, your file did not land in the document root, or your host is caching responses aggressively.
Common Pitfalls When Editing a WordPress robots.txt
A few patterns come up often enough to be worth naming:
- Treating it as access control. robots.txt is a public crawler request, not authentication. Any path you list is also visible to anyone who can guess the URL, and a malicious client can ignore the rules entirely. Use server-side authentication for private content.
- Blocking everything by accident. A stray Disallow: / rule under the wildcard group will hide the entire site from compliant crawlers. Allow all mode is the safer default until you know exactly which paths you want to hide.
- Capitalisation mismatches. RFC 9309 says crawlers compare paths case-sensitively and pick the longest matching rule. /Private and /private are different paths, and Disallow: /draft is not the same intent as Disallow: /draft$ (the trailing dollar sign anchors the match to the end of the path).
- Listing sensitive routes as a substitute for login checks. Putting /internal-finance/ into robots.txt hides the page from compliant crawlers but does not protect it from a logged-out visitor or a brute-force probe. Authentication belongs on the server.
- Assuming blocking removes URLs from search. Crawling and indexing are separate. A search engine may still list a URL it found through links and keep limited information about it without ever fetching the page. Use page-level indexing controls, HTTP headers, or a removal workflow for the actual goal.
Testing and Rolling Back Safely
Before you treat the new file as your default, run three quick checks. First, load the file in a private browser window and confirm the byte content matches what you generated. Second, hit one allowed URL and one disallowed URL through the search engine's URL tester (such as Google's URL Inspection tool) and confirm the crawler reports the same outcome you expect. Third, run your site through a robots.txt checker if you want to see how a wide set of crawler product tokens match your rules under RFC 9309 matching semantics.
If anything is off, restore the previous file. The generator never deletes the existing live file for you — that step is yours — so the rollback is simply uploading the backup you made in step 6 of the how-to. Once you are happy with the result, log the change in your deployment notes so the next person who edits the file knows what the baseline was supposed to look like.
The group and rule model used by the generator is defined in RFC 9309, Robots Exclusion Protocol, and the filename, location, and content-type expectations match Google's robots.txt creation guide.