No, llms.txt does not replace robots.txt or sitemap.xml. The three files solve different jobs on a public website. robots.txt expresses crawl preferences for participating crawlers, sitemap.xml inventories indexable URLs for search discovery, and llms.txt offers a short, curated Markdown overview of the highest-value resources for inference-time language model use. llms.txt is an emerging proposal documented at llmstxt.org and reference repositories, not an enforced web standard. It cannot grant access to blocked pages, override authentication, remove content from model training, or prove that any AI assistant will cite the linked material. Publishing it does not stop your existing robots.txt rules from applying, and search engines will continue to read your sitemap.xml on its own terms. The right mental model is additive rather than replacement: keep your robots.txt and sitemap.xml exactly as they are, treat llms.txt as a small editorial map beside them, and document that distinction so your team, your CMS author roles, and your future automation do not assume the new file retires the older ones.
llms.txt in Plain Language
An llms.txt file is a small Markdown document that follows a documented proposal: one required H1 containing the site or project name, an optional blockquote summary, optional explanatory details, and zero or more H2 sections. Each H2 groups Markdown list items whose core element is a link, optionally followed by a colon and a short note describing what the linked resource contains. The file is meant to be predictable for both language models and ordinary Markdown parsers, which is why the order is fixed: H1 first, then the optional summary, then any free-form details, then H2 link sections. A convention in the proposal reserves an H2 named "Optional" for secondary material that a consumer can skip when a shorter context is enough.
The point of the file is curation, not coverage. A long list of every page on a large site repeats the overload of a big sitemap and wastes the consumer's context window. The most useful roles to populate are canonical documentation, product explanations, policies, and stable reference pages; if the site reliably serves Markdown versions of those pages, link to those rather than to HTML pages that a tool would still have to convert. When in doubt, prefer the canonical, public, and stable over the new and the dynamic.
llms.txt is voluntary, emerging, and not a ranking signal. Publishing it does not force a crawler or assistant to request it, and no vendor has certified that the file influences search rankings or model behavior. That boundary is part of the proposal's honest framing and is the reason this article focuses on file roles rather than on promises.
robots.txt and What It Still Controls
robots.txt has been the standard way for decades to express crawl preferences. It sits at the site root, uses simple Allow and Disallow directives, and is read by compliant crawlers including the major search engine bots and many AI-adjacent crawlers. A well-formed robots.txt can block crawlers from specific paths, point them to a sitemap URL, and signal which user-agents the directives apply to.
What robots.txt does not do is at least as important as what it does. The file expresses preferences, not enforcement: crawlers that ignore the Robots Exclusion Protocol can still request any public URL, and robots.txt has no effect on a request that targets a URL it never blocks. It also does not delete pages from search indexes on its own, does not authenticate users, and does not describe the structure or content of resources. It is a polite request file with a long history, and it remains the right place to register crawl preferences.
For anyone publishing llms.txt, the implication is simple: keep your robots.txt intact. A new llms.txt file does not relax or replace any Disallow rule, and no inference-time consumer reading llms.txt can bypass robots.txt protections by referencing a page the file lists. Crawl control and resource map stay separate.
sitemap.xml and What It Still Does
sitemap.xml, governed by the sitemaps.org schema, is an XML inventory that helps participating search engines discover URLs on a site. A sitemap can declare lastmod timestamps, changefreq hints, and priority values, and it can be split into a sitemap index for very large sites. It is read by search engines as part of their normal discovery pipeline, and submitting one through search consoles is a routine SEO task.
As with robots.txt, the limits matter. A sitemap cannot force indexing, cannot influence ranking positions, and is not a substitute for internal linking or for content that genuinely earns inbound links. It also has nothing to say about content semantics or about which pages matter most to a human reader. A 50,000-URL sitemap that lists every product variant tells a search engine very little about which five pages an AI should treat as the canonical reference for the site.
That gap is exactly why a small, curated llms.txt file can sit beside a large sitemap.xml without conflict. The sitemap stays the inventory; llms.txt becomes the curated highlight reel.
How the Three Files Compare
The three files overlap only in the loose sense that they are all plain text documents served from a public URL. Their purposes, formats, and effects on what crawlers and models actually do are different.
| Concern | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Primary purpose | Crawl preferences for compliant crawlers | Inventory of indexable URLs for search discovery | Curated Markdown overview for inference-time use |
| Format | Plain text directives | XML following the sitemaps.org schema | Markdown: one H1, optional blockquote, optional details, ordered H2 sections |
| Can it block pages? | Yes, for participating crawlers | No, it lists rather than restricts | No, it points only to public content |
| Does it guarantee AI citations? | No | No | No, support is voluntary |
| Required placement | /robots.txt | Any URL listed from robots.txt or submitted to search consoles | /llms.txt or the documented subpath |
| Replaces another file? | No | No | Not a replacement; it complements both |
The table makes the replacement question easier to answer: no column wins the others. Each file addresses a job that the others do not attempt.
Build a Small llms.txt File: A Concrete Example
A working draft for a small documentation site can be assembled in a few minutes. The proposal is intentionally small, and a representative file keeps five curated references plus one genuinely secondary link moved to an "Optional" section, so the file reads as a curation exercise rather than an inventory dump. The file starts with a single H1 line containing the project name, such as "Acme Docs". Below the title comes an optional blockquote summary set off by a greater-than character that briefly states what the site covers. After the summary, two or three H2 sections group the links by purpose: a "Docs" section for canonical references, a "Policies" section for legal pages, and an "Optional" section for secondary material such as brand assets.
The exact order matters because the file is meant to be predictable for both language models and ordinary Markdown parsers. The H1 comes first, the optional blockquote summary follows when present, free-form details appear before any H2 sections, and the H2 link sections come last. Notes after a colon explain what each linked resource contains, which is the proposal's optional convention and helps consumers decide which links to fetch.
To produce the file in a deterministic way rather than hand-editing Markdown fragments, follow these concrete steps with a browser-side llms.txt generator:
- Enter the site or project name into the H1 field. This becomes the required top-level heading of the file.
- Add an optional blockquote summary that states in one sentence what the site covers. Leave it empty if a summary would only repeat the H1.
- Add zero or more H2 sections. Within each section, enter only canonical high-value links to public, stable pages. Skip pages that return errors or require authentication.
- For each link, include a short colon-prefixed note describing what the linked resource contains. Move any genuinely secondary resource into an H2 named "Optional".
- Generate the deterministic Markdown. Review the H1, summary, every section, every link, and every note before copying or downloading the file.
- Paste the draft into a separate validator to check the required H1, heading order, link-list syntax, duplicate targets, and bounded size. Treat the validator's output as concrete line-oriented issues rather than as vague warnings.
- Publish the validated file at /llms.txt or the documented subpath with a plain-text or Markdown-compatible content type, and periodically verify that every link still returns the intended content.
The generator does not crawl the site, does not assume canonical status, and does not promise that any consumer will read the result. It rejects unsafe or credential-bearing URLs, normalizes Markdown-sensitive text, and reports duplicate targets instead of silently multiplying entries. The editorial decision about which resources are essential still belongs to the author.
Publish, Validate, and Resist Overpromising
Once the draft is finalized, serve it at /llms.txt or the documented subpath with a plain-text or Markdown-compatible content type. Re-check every link by opening it in a fresh browser tab, confirm that notes match what each page actually contains, and confirm that no private, staging, or authentication-gated URL slipped into the list. A separate validation pass is worth the time: paste the draft back into a validator that checks the required H1, heading order, link-list syntax, duplicate targets, and bounded size, and treat its output as concrete line-oriented issues rather than as vague warnings. A reference implementation guide documents the same order and is a useful second opinion before deployment.
After publication, the work is not finished. The proposal and ecosystem can evolve, so any automation that depends on exact syntax should re-check the documented methodology. Periodically verify that each referenced page still returns the intended content and that no note has drifted away from what the page actually says. The single honest measurement is whether your own tools and workflows use the file; publication alone is not evidence of search or citation improvement, and treating it as such sets the team up for disappointment.
For readers who want a longer walkthrough of the build process, a step-by-step llms.txt walkthrough covers the same procedure in more detail, including how to validate the final draft before pushing it to production.
If you're weighing options, Meta Robots Generator on iPhone: A Safari Workflow covers this in detail.