Generate an llms.txt file only when you have a curated, stable set of canonical pages worth highlighting for language-model assistants, because llms.txt is an emerging proposal — not a ranking signal, not a crawl directive, not a sitemap replacement, and not a guaranteed discovery protocol. The file is a small Markdown index written in a deterministic order: one required H1 with the site or project name, an optional blockquote summary, optional non-heading details, and zero or more H2 sections that group Markdown link items, each with an optional colon-separated note. Because every interacting element changes whether the file parses correctly, the decision to publish one should rest on editorial criteria — do you have canonical pages worth curating, and will you keep them accurate — rather than on any promise of citation, ranking, or crawler adoption.
That definition matters because the same search that leads you to an llms.txt example often arrives with a hidden assumption: that publishing the file will be visible to search engines or AI assistants on its own. The honest answer is that the proposal is voluntary. Crawlers and assistants are not obliged to request /llms.txt, follow its links, or treat it as authoritative. Some sites ship one as a courtesy to model providers; a few crawlers have begun to fetch it experimentally. The file complements, rather than replaces, robots.txt, sitemap.xml, and structured data, each of which serves a different job. Knowing that boundary is what turns the decision from "should I add another SEO file?" into "do I have a small, stable, high-value reading list worth maintaining for inference-time use?"
The Role and Limits of llms.txt
The llms.txt proposal, documented at llmstxt.org and referenced in community implementations such as the AnswerDotAI llms-txt reference repository, defines a predictable Markdown order for a curated overview. The H1 is mandatory and contains the project or site name. A blockquote summary can follow when a one- or two-sentence framing helps. Free-form details appear before any H2 link sections. Each H2 heading groups Markdown list items whose required core is a link; an explanatory note can follow a colon. A section named Optional has a defined convention: links under it identify secondary material that a consumer can skip when working with a shorter context window.
What the proposal is not is just as important. llms.txt is not a robots directive — it does not restrict crawlers. It is not a sitemap — it does not enumerate indexable URLs for search discovery. It is not a structured-data artifact — it does not describe entities on a page. It does not control training data, and it does not force a model to cite the publishing site. The proposal is an editorial surface, and the discipline of curation matters more than length: a file that mirrors a sitemap and lists every URL wastes the consumer's context rather than guiding it.
Decision Criteria: When Generating llms.txt Makes Sense
Whether you should publish the file comes down to four questions you can answer with a short editorial pass.
- Do you have a small, stable set of canonical resources? Documentation hubs, product explanations, policies, and reference pages are natural candidates. A long marketing blog with thousands of posts is usually a poor fit because curation is hard to enforce.
- Can you keep the list current? Stale links are a worse signal than no file. Plan for periodic verification of every URL listed under each section.
- Is inference-time discovery worth your time? If you build internal tools, assistants, or pipelines that read /llms.txt, the file pays back directly. If you are optimizing purely for traditional search, sitemap.xml and structured data carry the load.
- Does the file complement — not duplicate — existing controls? robots.txt, sitemap.xml, and JSON-LD each have their own job. llms.txt belongs alongside them, not in place of any of them.
If two of those four answers are clearly yes, a curated file is probably worth your time. If only one is yes, defer it. The cost of publishing a half-maintained llms.txt is higher than the cost of not publishing one.
Where llms.txt Fits Next to robots.txt and sitemap.xml
The table below summarizes the role each file plays in a healthy site so the decision is not framed as llms.txt versus the others. Each row describes the file's purpose, its mechanism, and the audience it serves.
| File or surface | Purpose | Mechanism | Primary audience |
|---|---|---|---|
| robots.txt | Express crawl preferences for participating crawlers | Exclusion rules applied per user-agent | Search and AI crawlers that respect the standard |
| sitemap.xml | Inventory indexable URLs for search discovery | XML list of canonical pages with optional metadata | Search engines and supporting tools |
| Structured data (JSON-LD, Microdata) | Describe entities and page content | In-page markup parsed by crawlers and parsers | Search engines, knowledge panels, schema parsers |
| llms.txt | Curate a small Markdown overview and high-value references for inference-time use | Predictable H1, optional summary, ordered H2 sections with link items | Language-model assistants and human readers |
The same lesson applies to llms-full.txt, the longer companion file proposed at llmstxt.org that bundles substantive content into a single context-friendly payload. Whether you also publish /llms-full.txt depends on whether you can vouch for the licensing, accuracy, and token cost of bundling full content. Treat it as a separate editorial decision.
Walk Through an llms.txt Example With the Generator
Once you decide to publish the file, the practical task is to produce a draft in the proposal's documented order, review it, and ship it. The llms.txt Generator is a local-only form that serializes the documented Markdown order without crawling your site. The example below uses a fictional documentation site called Acme Docs to make the structure concrete.
- Enter the site or project name. Type the exact name you want as the H1. The generator treats this as the required first element; a missing or empty title is rejected.
- Add an optional summary. One or two sentences framed as a blockquote. For Acme Docs that becomes a quote that reads: Acme Docs is the reference manual for the Acme widget API, configuration, and CLI.
- Add optional non-heading details. Short paragraphs that explain scope, versioning, or audience. They appear before any H2 sections and must not contain heading characters that would create duplicate top-level headings.
- Group canonical links under clear H2 sections. Place each Markdown link as a list entry. Only http and https URLs are accepted; credentials, javascript, or data schemes are rejected before serialization. Labels, notes, and summaries are normalized so control characters or accidental Markdown delimiters cannot break the structure. Duplicate normalized URLs are flagged instead of being multiplied.
- Decide what belongs in an Optional H2 section. The Optional section has a defined convention for secondary material that a consumer may skip in a shorter context. The generator does not auto-label a resource as optional; that decision is editorial.
- Generate, then review the Optional section and every note. The output is deterministic plain text. Copy the result, or download a file named llms.txt. Re-read it as content rather than as a mechanical SEO artifact — confirm the title and summary match the site, every link is canonical, notes are factual, optional resources are genuinely secondary, and no private or staging URL slipped in.
A trimmed illustration of the kind of file you may produce for Acme Docs reads top to bottom in this order: Acme Docs as the H1, a blockquote summary explaining the scope, a short details paragraph noting the release cadence, an Getting Started H2 with two link items (a quickstart and an authentication page, each with a colon-separated note), a Reference H2 with one link item (the CLI reference), and finally an Optional H2 with a changelog link that consumers may skip when working with a shorter context window. Re-reading the file as content is where the editorial pass happens.
Validate the Draft Before Publishing
Validation is a separate mode in the same tool. Paste an existing llms.txt draft to check the required H1, heading order, link-list syntax, duplicate targets, and bounded size. It reports concrete line-oriented issues and warnings, but it does not rewrite the file, follow any listed URL, or claim the linked resource is accurate. Crucially, validation success means the draft matches the tool's reading of the current spec — it does not certify broader vendor support. The proposal and ecosystem can still evolve, so recheck the methodology and source links before you build automation that depends on exact syntax.
Publish, Maintain, and Measure Honestly
Publish at the path you intend to support — typically /llms.txt, or a subpath if the file applies to a subsection — with a plain-text or Markdown-compatible content type. Then measure the right things. Server logs may reveal occasional fetches from AI-associated user agents, which is evidence of access but not of citation. Citation, ranking, and training behavior are not promised by the proposal. The honest metrics to track are whether your own tools, assistants, or pipelines read the file, whether the curated links remain canonical, and whether the notes still match what each linked page actually contains.
For a workflow that emphasizes review before deployment rather than generation alone, the llms.txt generation with editorial review guide walks through the same generator with an extra editorial pass. If you want a side-by-side look at how the file relates to existing crawler controls, the llms.txt versus robots.txt and sitemap.xml comparison covers that decision explicitly.
Periodically re-verify each referenced page. The proposal is small, the file is short, and the maintenance burden is editorial — not algorithmic. If the curation stops being honest, take the file down rather than let it drift.