An llms.txt file is a small, curated Markdown overview of a site, structured as one required H1 title, an optional blockquote summary, optional explanatory details, and zero or more H2 sections of grouped links in a predictable order. The format originates from the llmstxt.org proposal and is meant to give language models a short, human-readable briefing about a project before they consider the deeper content. Crucially, the file is an emerging proposal, not a guaranteed discovery protocol, a robots directive, or a sitemap replacement. A site can publish one to make its canonical resources easy to find, but no crawler is forced to read it, and there is no promise of ranking or citation lift. What the file does well is communicate editorial intent: which pages best represent the product, which references are secondary, and which sources are safe to surface for inference-time use. A well-built llms.txt is shorter and more selective than a sitemap, and that brevity is the point — it is a curated map, not a complete index.
What an llms.txt File Contains (and What It Doesn't)
The proposal specifies a tiny grammar. The only required element is a single H1 containing the project or site name. After that, the file may include an optional blockquote summary, optional non-heading details, and zero or more H2 sections of Markdown list items. Each list item's required core is a link; an optional colon and note can explain what that linked resource contains. Order matters because the file is intended to be predictable for both language models and ordinary parsers, so the generator enforces the documented sequence rather than accepting arbitrary Markdown fragments that could produce duplicate top headings or misplaced sections.
A defined convention also exists for a special H2 named Optional. Links placed under that heading identify secondary material that can be skipped when a consumer needs a shorter context. The decision of what counts as optional is editorial: the author chooses which sources are essential for understanding the site and which can be safely dropped under load. The tool does not auto-label a resource as optional — that is a curation call, and it is one of the most important decisions in the file.
Equally important is what the file does not do. llms.txt is not a robots directive, not a security control, not a sitemap replacement, and not a guaranteed discovery protocol. Publishing it cannot grant access to blocked pages, override authentication, remove content from model training, or prove that an AI answer will cite the site. It complements existing web controls rather than replacing any of them.
How to Create the File Step by Step
Use the llms.txt Generator as a transparent starting point for assembling the file in the documented order. The form runs entirely in the browser, accepts only the inputs the proposal defines, and produces a deterministic plain-text output you can copy or download without sending anything to an external service.
- Open the llms.txt Generator and enter the site or project name in the required H1 field. Add an optional blockquote summary if you want a one-line description, and add free-form details if you need background that does not belong in a list.
- Add only canonical, high-value links under clear H2 section headings. Reserve one section labeled Optional for genuinely secondary material that can be skipped under a tight context.
- Generate the deterministic Markdown, then read the Optional section and every note to confirm they match the linked pages. Edit labels and wording inside the form if anything reads wrong before downloading.
- Copy the generated text or download the file named llms.txt, then open the saved file and review it as content. Confirm the title and summary match the site, every link is canonical, notes are factual, optional resources are genuinely secondary, and no private or staging URL appears.
- Paste the final edited draft into validation mode to check the required H1, heading order, link-list syntax, duplicate targets, and bounded size. Treat the validator as a linter for the documented interpretation, not an authority on broader vendor support.
- Publish the validated file at the intended path, typically /llms.txt, served with a plain-text or Markdown-compatible content type.
- Periodically verify each linked page still returns the intended content and re-validate the file after every edit. Recheck the proposal's source pages before building any automation that depends on exact syntax.
What the Validator Checks in a Draft
Validation is a separate mode from generation. Paste an existing llms.txt draft to check the structural interpretation the tool applies to the proposal. The validator reports concrete line-oriented issues and warnings rather than silently rewriting the file. Specifically, it checks for the presence of the required H1, the correct heading order, valid Markdown link-list syntax, duplicate normalized URLs, and a bounded total size.
URLs are reviewed before serialization. The generator accepts HTTP and HTTPS links and rejects credential-bearing or executable/embedded-data schemes. Labels, notes, headings, and summaries are normalized so control characters or accidental Markdown delimiters cannot break the generated structure. Duplicate URLs are reported rather than silently multiplying entries, which keeps the curated map honest. The validator never follows any listed URL and never claims the linked resource is accurate — verifying the destination is the curator's job, and the canonical examples in the llms-txt reference repository maintained by Answer.AI are a useful sanity check for your own drafts.
Validation success means the draft matches the tool's documented interpretation of the current proposal. It is not a certificate of broader vendor support, because the ecosystem is still evolving. Treat it as a reliable local linter, then re-validate whenever you change the workflow or upgrade the tool.
Where to Publish and How to Maintain
The conventional location is /llms.txt at the site root, served with a plain-text or Markdown-compatible content type. Subpaths are acceptable when the file is scoped to a documentation section, but the root path is what most consumers expect by default. After publishing, treat the file as living content rather than a one-off deployment artifact.
Maintenance is straightforward and editorial. Each linked URL needs to keep returning the intended content; a reference that 404s or returns the wrong page quietly undermines the file's value. The optional section should remain a deliberate subset of secondary material, not a dumping ground for everything that did not make the primary list. Re-validate the file after edits so heading order, link syntax, and duplicate handling stay correct.
Two limits are worth naming explicitly. First, publishing llms.txt does not force a crawler or assistant to request it. Second, the proposal and ecosystem can evolve, and the methodology references visible to the generator are the canonical source of truth. Recheck them before you build automation that depends on exact syntax, and re-validate whenever you upgrade the workflow.
How llms.txt Compares to Other Site Files
llms.txt overlaps with several familiar web controls in name only. The roles are distinct, and a healthy site can use all of them together without conflict. The table below summarizes the documented scope of each file type so you can place llms.txt correctly in a normal SEO stack.
| File or signal | Documented purpose | Read by | Effect if omitted or absent |
|---|---|---|---|
| llms.txt | Curated Markdown overview with optional summary and H2 link sections, per the llmstxt.org proposal. | Participating AI assistants and tools that opt in. | No forced effect. Models and assistants are not required to fetch or use it. |
| robots.txt | Express crawl preferences for participating crawlers, per RFC 9309 and vendor extensions. | Web crawlers that choose to honor it. | Crawlers may index any non-restricted path; stated preferences are not enforced. |
| sitemap.xml | Inventory indexable URLs for search discovery and freshness signals. | Search engines and consumers of the sitemap protocol. | Discoverability still works via links; the explicit URL list is simply missing. |
| Structured data (JSON-LD, Microdata) | Describe entities, attributes, and page content for rich results and entity graphs. | Search engines and consumers of schema.org vocabularies. | Rich-result eligibility is reduced; entity understanding is implicit only. |
Curation Beats Length
The biggest design choice in an llms.txt file is what to leave out. A file that lists every page repeats the overload of a large sitemap and wastes a consumer's context window. The proposal's intent is the opposite: a short, opinionated map that points a model at the right handful of pages to read first. Prefer canonical documentation, product explanations, policies, and stable reference pages. If your site reliably serves Markdown versions of its pages, link to those; if it does not, do not invent .md URLs that return errors, because broken references undermine the file.
The llms.txt Generator never scans a domain or sitemap, and that is a deliberate boundary. A purely client-side form cannot prove which dynamic pages are canonical, current, accessible, or safe to recommend. The author remains responsible for selecting sources and confirming that each public URL returns the intended content. After download, read the file once as if you were the model: does the title match the site, do the notes still describe the linked pages, and is anything in the Optional section actually optional? If the answer is yes, the file is doing its job — and that is the right place to stop measuring.