Shopify serves /robots.txt from the robots.txt.liquid template inside your active theme, so creating a robots.txt file in Shopify means drafting a standards-aligned file in a browser-side generator and pasting the reviewed text into the theme editor rather than uploading anything through FTP. Every Shopify store already responds at /robots.txt because the platform emits a default file when no template is present, but that default block-list may not match what you actually want search engines to see. Producing a custom file starts with three inputs — the store's HTTP or HTTPS origin URL, an overall crawler policy, and a list of disallowed paths if you pick selective mode — and ends with a small UTF-8 text file you copy into the theme template. The result still has to govern the exact scheme, host, and port of the store, and every listed path will be public, so the goal is a clean, reviewable file that matches what compliant crawlers should follow.
Why Shopify Handles Robots.txt at the Theme Layer
Shopify does not let you drop a robots.txt file directly into the store root through SFTP or the Files API. The platform owns that location and serves /robots.txt from a theme template, which is why the official workflow routes through the theme editor. In the Shopify admin, the route is Online Store → Themes → … → Edit code → robots.txt.liquid (or a section in theme settings on newer themes). Whatever text you save there becomes what crawlers receive at /robots.txt.
This shape matters because any file the generator produces has to be pasted into that template, not uploaded to the root. A file sitting on a different subdomain, on http when your store is https, or under a subdirectory does not govern the same origin. If you draft rules in the generator using http://yourstore.myshopify.com but the live store serves https://www.yourstore.com, the polished text still has to be reviewed and the sitemap line edited to match the canonical host you actually want governed.
The generator does not connect to the live site, fetch the existing file, or confirm that a crawler has refreshed its copy — those steps all happen through the Shopify admin and the search engines' own tools. Treat the generator as a drafting surface that produces clean text, and treat the theme editor as the place where that text actually goes live. For broader reading on what robots.txt is meant to do, the practical guide to creating a robots.txt file covers the protocol in plain English.
Drafting the File With the Robots.txt Generator
The Robots.txt Generator builds a standards-aligned robots.txt file entirely in the browser. You enter the full store URL (http or https, with an explicit port if non-standard), choose one of three policy modes, and the tool returns a small text block ready to download. In selective mode you supply one path per line — paths must begin with a slash, identical duplicates are dropped in first-seen order, and the list tops out at 50 rules. The output is plain UTF-8 text designed to be served as text/plain at the exact origin you entered.
Three policy modes are available, and each writes different lines:
| Mode | What the generator writes | When it fits a Shopify store |
|---|---|---|
| Allow all | User-agent: * followed by Allow: / | When you want compliant crawlers to reach every public route and rely only on page-level indexing controls. |
| Selective disallow | User-agent: * followed by one Disallow line per path | When you want search engines to skip /cart, /account, individual collection filters, or staging paths while indexing the rest of the catalog. |
| Block all | User-agent: * followed by Disallow: / | Almost never — only when you are intentionally pulling a store offline or staging a full no-index behind authentication. |
A final Sitemap: line is appended automatically, pointing at the normalized origin plus /sitemap.xml. If your store's sitemap actually lives elsewhere, replace that line before publishing — the generator cannot know about a custom path on its own. A page path, query, or fragment typed into the URL field is stripped back to its origin so the default sitemap does not accidentally inherit an unrelated route, while an explicit port is preserved.
Generate Your Shopify Robots.txt Step by Step
Here is the focused sequence that turns a store URL into a Shopify-ready robots.txt.
- Open the Robots.txt Generator and enter your full store origin in the URL field (for example https://www.yourstore.com). Use the exact scheme a real visitor would use; an explicit port stays in the result if you need one.
- Pick a policy mode. Choose allow all when no paths should be hidden from compliant crawlers. Choose selective disallow if you have specific routes to exclude. Choose block all only when the entire site should be off-limits to compliant crawlers.
- If you chose selective disallow, list the disallowed paths on separate lines, beginning each with a slash. Examples that often matter on a Shopify catalog: /cart, /checkout, /account, /search, /collections/*filter*, /pages/private, plus any staging or preview paths you do not want indexed. Wildcard asterisks and trailing dollar-sign anchors are preserved, case is meaningful, and identical duplicates are removed.
- Click generate and read the resulting text. Confirm the User-agent line reads User-agent: *, that Allow or Disallow lines use the policy you picked, and that the Sitemap line points at the right URL.
- Download or copy the produced text. Compare it line by line with the current robots.txt served by your Shopify store (visit /robots.txt in a browser to read what is live). Preserve any intentional crawler-specific groups the generator did not emit, and keep a copy of the old file for rollback.
- Paste the reviewed text into the theme template in your Shopify admin (Online Store → Themes → … → Edit code → robots.txt.liquid) or the theme settings section if your theme exposes one. Save the changes.
After saving, hard-refresh your browser at https://www.yourstore.com/robots.txt and confirm the live text matches what you intended. The generator does not push to the store, does not submit to a search engine, and does not invalidate caches — those confirmations are manual.
A Closer Look at RFC 9309 Rules in Your File
The output follows the group-and-rule model standardized by RFC 9309, which formalized the Robots Exclusion Protocol. One group begins with User-agent, followed by Allow or Disallow rules. The wildcard product token (*) applies when a crawler has no more specific matching group, which is why the generator emits a single general group rather than pretending to manage every search engine's private extensions.
Four protocol behaviors will quietly change the meaning of what you typed, and the generator already enforces them:
| Mechanism | What RFC 9309 says | What to watch for in a Shopify file |
|---|---|---|
| Case sensitivity | Paths compare case-sensitively. | /Private and /private are different routes; match the casing used by your live URLs. |
| Longest match wins | Among matching rules, the longest pattern is used. | Disallow: /draft and Disallow: /draft$ both work but mean different things; the second restricts to an exact match. |
| Wildcards | * matches any sequence of characters. | Disallow: /search catches /search?q=red and the bare /search path; the $ anchor restricts to exact matches. |
| Path start | Rules match from the first character of a URL path. | Every rule must begin with a /; lines that start with anything else are invalid and the generator rejects them. |
The generator also strips control characters, rejects hash signs so a path can never accidentally turn into a comment, caps the selective list at 50 unique rules, and removes exact duplicates in first-seen order. Crawl-delay is omitted on purpose because it is not part of RFC 9309 and is not supported consistently across crawlers. If you need crawl rate hints, configure them in the relevant search engine's tools rather than embedding them in robots.txt. For Google-specific publishing notes, the Google crawling infrastructure documentation on robots.txt walks through hosting requirements and edge cases.
Publishing the File to Your Shopify Theme
Once the generated text passes your review, the publishing step happens entirely inside Shopify. The platform does not let you upload a robots.txt file to the root with FTP, and the file the generator produces must end up in the theme template the store actually serves. Open Online Store → Themes in the admin, click … on the published theme, choose Edit code, and look for robots.txt.liquid. Paste the reviewed text in, save, and re-check /robots.txt in a fresh browser tab.
Newer themes sometimes expose a robots.txt section inside theme settings instead. Functionally the file is identical; the only difference is the editing surface. If neither surface is visible, the published theme may not yet have a robots.txt.liquid template — in that case, add the file from the templates area of the code editor (using the file extension the theme expects, .liquid, not .txt), then paste the generated text in.
The file must be served at the exact scheme, host, and port that it governs. A staging domain on http does not control the production https storefront, and a draft file used on a subdomain does not govern the apex. If you operate more than one storefront, generate a separate file per origin — the tool enforces one origin per run, so reusing text from a draft-store URL on a production store will silently change the sitemap line and the scope of every rule.
Review and Verification Before Going Live
Three quick checks catch most mistakes before they ship. First, read the file you are about to publish and confirm every Disallow path uses a real route on the live store — a typo in a path quietly leaves the route open. Second, drop one public URL and one blocked URL into the relevant search engine's URL tester where available to see whether the crawler reaches each one after the change. Third, keep the previous robots.txt text in a separate note so you can paste it back if the new file breaks indexing of an important page.
Robots.txt is not authentication or authorization, and even compliant crawlers may refuse to honor a blocking request. RFC 9309 explicitly says these rules are not access authorization: a malicious client can ignore them, and every listed path is publicly visible because the file itself is public. If a route is sensitive, gate it behind server-side authentication rather than relying on a Disallow line. Crawling and indexing are also separate concerns, so blocking a path stops fetching but does not guarantee removal from search results — for that, use page-level meta robots, X-Robots-Tag headers, the search engine's removal tools, or authentication as the actual goal requires.
For ongoing maintenance, treat the file as something you edit deliberately. When you add staging paths, query-string-heavy collection filters, or seasonal landing pages you do not want indexed, return to the generator, update the policy, regenerate, and compare. Each generated text stays in the current tab as long as the page is open; once the browser tab is closed, only the file published to the theme is the source of truth, so a quick "last working" copy somewhere safe is the cheapest insurance you can keep.