Does WordPress create a robots.txt automatically?

Yes. WordPress renders a virtual robots.txt at /robots.txt through its WP_Rewrite endpoint even when no file is on disk. The default contains User-agent: *, Disallow: /wp-admin/, and Allow: /wp-admin/admin-ajax.php. Uploading a real file at the document root overrides the virtual response.

Can I edit the robots.txt file from inside the WordPress dashboard?

Only if a SEO plugin exposes an editor. Yoast SEO, Rank Math, and All in One SEO all add a settings screen that overwrites the virtual response with your pasted rules. Without such a plugin, the core admin dashboard has no built-in robots.txt editor.

Will blocking a URL with robots.txt remove it from search results?

Not necessarily. Crawling and indexing are separate steps: a search engine can still discover and list a URL through inbound links and retain limited information such as a title or snippet, even when the page is not fetched. For real removal, use a noindex header, the search engine's removal tool, or password protection.

Is robots.txt the same as access control or authentication?

No. RFC 9309 explicitly says robots.txt is a public crawler request, not a security mechanism. Disallow rules are honored by cooperative clients and ignored by anyone who reads your file directly. Sensitive content in WordPress needs server-side authentication, role checks, and page-level indexing controls, not a path blocking rule.

Does WordPress overwrite my uploaded robots.txt after an update?

No, the core never deletes or modifies a physical robots.txt when it updates. The virtual response only appears when no physical file is present. Plugins can write or modify a physical file, but most SEO plugins instead overwrite the virtual response, leaving your uploaded file untouched.

How to Create a Robots.txt File in WordPress

In WordPress, a robots.txt file at the site root tells compliant crawlers which paths they may request, using Allow and Disallow rules standardized in RFC 9309. The rules live in a single plain-text UTF-8 document with one user-agent group, path patterns matched from the URL's first character, and case-sensitive longest-match resolution. WordPress makes this slightly unusual because even when no physical file exists, the application serves a virtual robots.txt generated from PHP, so most sites already respond at /robots.txt with a permissive default. To take real control, you generate a standards-aligned file in the browser, replace or augment that virtual response, and confirm the deployed version is what loads. The Robots.txt Generator handles the protocol syntax, the case sensitivity, the longest-match behavior, and the origin-relative Sitemap line, so the file you publish stays inside RFC 9309's documented shape. WordPress then exposes two publishing paths: writing the file directly into the site root, or editing the rendered virtual file through a SEO plugin. Both end with the same goal, a real robots.txt at the exact scheme, host, and port you intend to govern.

how to create robots txt file in wordpress

Why WordPress Robots.txt Is Different From a Static Site

Static sites either ship a robots.txt file at the root or they do not. WordPress never has to be that binary because the application itself renders one. WordPress registers a virtual /robots.txt endpoint during initialization through WP_Rewrite, so a request to that path produces a dynamically generated response even when no file is on disk. The default virtual response includes three rules:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This default is permissive enough to keep a WordPress site crawlable while keeping the admin out of search indexes, and it is fine for many sites. It also limits you: there is no admin screen for adding custom rules, and any time you want to extend the file you need either a plugin or a physical file. SEO plugins such as Yoast SEO, Rank Math, and All in One SEO add a settings tab that overwrites the virtual response with whatever you paste. The result is still a virtual file; the plugin is just translating your edits into the same dynamic endpoint.

Uploading a real robots.txt to the document root takes a different code path. Apache and Nginx serve the file directly and never hand the request to WordPress, so your file wins over the virtual one automatically. WordPress sites installed at the document root keep robots.txt in the same folder as wp-config.php. Sites hosted in a subdirectory and reachable at the apex still publish the file alongside the install at the top level of whatever origin you actually want to govern, which must be the exact scheme, host, and port the rules apply to.

Plan Your Disallow Paths Before You Generate the File

Open a text file and write down what you want hidden from compliant crawlers before you open a generator. The choices depend on what your WordPress installation is actually serving. Typical candidates include:

/wp-admin/ (admin area, including edit screens)
/wp-includes/ (core files rarely useful in search results)
/search (internal search result pages)
/?s= (same as above via query parameters)
/cart/, /checkout/, /my-account/ (WooCommerce flows)
/tag/ (tag archives that often add no SEO value)
/author/ (author archives for single-author sites)
/feed/ (RSS feeds)
/trackback/ (legacy trackback endpoints)

Skip paths that do not exist on your site. A Disallow rule on a path that returns 404 does nothing visible and adds noise. Also skip sections you actually want indexed; blocking your real content category by accident is the most common WordPress robots.txt mistake.

For each path, decide whether you want prefix matching or exact matching. RFC 9309 uses longest-match, so Disallow: /draft blocks /draft, /drafts, and /draft/secret the same way. To match only the exact segment, append a dollar sign: Disallow: /draft$. Paths are case-sensitive: /Private and /private are separate rules. The official guidance from Google's robots.txt reference underlines the same longest-match and case-sensitivity rules for compliant crawlers.

Plan the Sitemap line at the same time. The generator appends Sitemap: <your origin>/sitemap.xml automatically, where the origin is the scheme plus host plus any preserved explicit port. If your real sitemap lives elsewhere, or you maintain a sitemap index with several files, edit the generated output before publishing. Every Sitemap directive must be a full absolute URL, not a path, and on its own line.

Generate the Robots.txt File in Your Browser

The generator follows the protocol's group-then-rules structure and runs entirely in your tab, so the URL you enter never leaves the browser. The steps mirror the tool's documented workflow:

Open the Robots.txt Generator.
Enter the complete site URL in the URL field, including the scheme (http or https) and any non-default port. URLs with a path, query, or fragment get reduced to the origin, so pasting https://example.com/blog/?utm=1#top produces the same basis as pasting https://example.com. Explicit ports are preserved exactly.
Choose a crawler policy. Allow all crawlers writes Allow: /, right for public content sites that want maximum indexing. Block all crawlers writes Disallow: /, right only for staging environments or private sites that no compliant search engine should touch. Selective disallow is the common WordPress choice because it lets you list admin and search paths while keeping posts and pages crawlable.
In selective mode, enter one path per line. Each non-empty rule must start with /. Wildcard * and end-anchor $ are allowed because the protocol defines them as matching characters. Hash signs #, which the protocol treats as comments, are rejected so a stray character does not make part of your rule vanish. Exact duplicates collapse to the first occurrence, and the list caps at 50 unique rules.
Generate the file. The output is plain UTF-8 text containing User-agent: *, the rules you selected, and a Sitemap line based on your origin.
Inspect the generated output. Confirm capitalization, prefixes, and anchors. Confirm the Sitemap line points to a real sitemap; if your sitemap is at a different path or you maintain a sitemap index, edit the text before publishing.

If you want to compare policy shapes side by side, this table summarizes what each option actually writes:

Crawler policy	Generated rule block	Typical WordPress use	Sitemap line
Allow all crawlers	Allow: /	Public content sites seeking full indexing	Always appended
Selective disallow	One Disallow line per entered path	Most WordPress blogs and business sites	Always appended
Block all crawlers	Disallow: /	Staging, pre-launch, or private intranets	Always appended

Publish the File on WordPress

Pick one publishing path, not both, to avoid version conflicts. Two reliable routes exist.

Route A: upload a physical robots.txt. Use the host's file manager (cPanel, Plesk), an SFTP client, or the host's file browser. Navigate to the document root of the WordPress install. Create a new file named robots.txt in lowercase. Paste the generated text and save, served with the correct MIME type text/plain. WordPress is now bypassed for that path; the server returns your file as static content directly. To roll back, delete the file and the virtual response returns automatically.

Route B: edit through a SEO plugin. Install a reputable SEO plugin if you have not already, then open its robots.txt editor. Yoast offers it under SEO → Tools → File editor, Rank Math under General Settings → robots.txt, and All in One SEO under Tools → robots.txt. Paste the generated text into the plugin's editor and save. The plugin injects your content into the virtual response on every request. This route is faster, but it ties you to the plugin: disabling the plugin reverts the file to WordPress's default virtual response.

Note that the two routes do not compose cleanly. A SEO plugin's editor cannot write a physical file for you, and a physical file wins over the virtual response if both exist, which means plugin edits silently disappear. Once you pick one, stick with it.

For adjacent WordPress work, the practical guide to creating a robots.txt file covers the protocol in more detail. For sitemap-specific WordPress decisions, building the file by hand is covered under creating an XML sitemap in WordPress without plugins.

Verify and Maintain the File After Publishing

Always run the same four checks after the file lands.

Direct load. Open https://yourdomain.com/robots.txt in a browser. Confirm the response body matches what you generated and confirm the Content-Type is text/plain. A Content-Type of text/html usually means a theme or template is intercepting the request and serving the wrong document.

Search-engine testers. Submit the URL in the search engine's own tooling where available. Google's Search Console exposes a robots.txt Tester that reports the version Googlebot last fetched. Bing and Yandex offer similar checks for compliant crawlers they manage.

Spot-test important URLs. Pick one URL you want indexed and one you want blocked, then simulate a compliant fetch. The path matching should agree with your intent and follow longest-match rules, per the standardized behavior in RFC 9309.

Keep the old file. Save a copy of what you replaced. If a search engine sees a regression, restore the old file first and then debug.

Do not treat robots.txt as access control. As RFC 9309 says explicitly, the protocol is not a security mechanism: compliant crawlers may honor the requests, malicious clients can ignore them, and every path you list is publicly visible. A draft, a private post, a paywalled article, or a membership section still needs server-side authentication and authorization. The same Disallow that hides /drafts/ from crawlers does not hide it from anyone who guesses the URL.

Block when crawling adds no value (thin tag archives, internal search results, admin endpoints). Do not block when the goal is real protection; the right tool there is authentication, not a robots.txt rule. Update the file when the content model changes: tag slugs, author archives, custom post type archives, and WooCommerce endpoints all shift over time. Re-run the generator and republish whenever the structure changes, and remember that Crawl-delay directives are intentionally not added because they are not part of RFC 9309 and are not supported consistently across crawlers.

Finally, do not assume blocking guarantees removal. A URL blocked by robots.txt can still surface in search results with limited information if other sites link to it. For actual delisting, use the search engine's removal tool, an X-Robots-Tag noindex header, or password protection for the page itself.