How to Create an llms.txt File for Your Website

An llms.txt file is a plain-text Markdown document placed at the root of a website that lists the site's name, an optional summary, and a curated set of canonical links grouped under section headings, so language models and AI assistants have a predictable map of the most useful public pages. The format proposed at llmstxt.org requires exactly one H1 containing the project or site name, then permits a blockquote summary, free-form details, and zero or more H2 sections whose list items each include a link, with an optional note after a colon. An llms.txt file is not a robots directive, a sitemap, or a security control; it does not force crawlers to fetch the file, does not block or grant access to any page, and does not guarantee that an AI answer will cite the site. It is an editorial artifact, and the value comes from careful curation rather than from automated crawling.

Because the proposal is still evolving, the practical approach is to write the file yourself, generate a draft with a tool that respects the documented order, validate it, and publish it without overpromising what it will do. The rest of this guide walks through the structure, the curation decisions, and the validation and publishing steps that turn a Markdown draft into a maintainable llms.txt.

What an llms.txt File Actually Does

llms.txt is a proposal for a small, curated Markdown overview of a site that can be read by language models during inference. Think of it as a reading list rather than an instruction set: the file tells an AI which pages on the site are worth reading when a user asks a question the site could answer. Each entry is a link plus, optionally, a short note explaining what the linked page contains. The proposal calls out an "Optional" section whose links identify secondary material that a model can skip when it only needs a shorter context.

The file does not replace the controls you already use. robots.txt expresses crawl preferences for participating crawlers. A sitemap.xml file inventories indexable URLs for search discovery. Structured data describes entities and page content. llms.txt sits beside those mechanisms and offers a different signal: a hand-picked set of high-value references chosen for inference-time use. Each mechanism keeps its own purpose, and the file should complement the rest of your SEO setup rather than substitute for it.

The Required Structure of an llms.txt File

The proposal defines a strict top-to-bottom order, and the value of the format depends on that order being predictable. The required elements, in sequence, are:

1. A single H1 line containing the project or site name. This is the only required element.
2. An optional blockquote summary, placed directly under the H1, that gives a one-paragraph description of the site.
3. Optional free-form details, written as plain Markdown paragraphs or lists, that explain anything a reader needs before the link sections begin.
4. Zero or more H2 sections, each containing Markdown list items whose required core is a link. An optional colon followed by a note can describe what the linked page contains.
5. An H2 section literally named "Optional" whose links identify secondary material that a model can skip when it needs a shorter context.

The H1 must come first, the summary follows when present, free-form details appear before any H2, and the H2 sections come last in the order you choose. The AnswerDotAI llms-txt reference repository demonstrates this layout with real-world examples. Because parsers and language models both expect this order, an llms.txt file with two top-level headings, misplaced summaries, or notes that contain Markdown delimiters can break the structure silently.

A few conventions worth noting:

• The proposal allows a single H1, so duplicate top-level headings are an error rather than a stylistic choice.
• The "Optional" section has a defined meaning; the generator does not automatically label a resource optional, since that is an editorial decision.
• URLs with credentials or executable schemes are unsafe and should not appear in the file at all.

How to Build Your llms.txt File

The llms.txt Generator builds a draft locally from the form fields you fill in, then lets you validate the final edited file against the documented Markdown order. Nothing is crawled, and the form values are not sent to an external service.

1. Enter the site name, optional summary, and details. The site name becomes the required H1. The optional summary, when included, is wrapped as a Markdown blockquote directly under the H1. Any additional paragraphs or short lists belong between the summary and the first H2 link section.
2. Add only canonical, high-value links under clear section headings. Group resources into H2 sections such as Docs, Products, Pricing, or Policies. Use the section literally named "Optional" for secondary material that a shorter context can skip. Each list item must include a link, and a short note after a colon is allowed.
3. Generate the deterministic Markdown and review it. The tool produces plain text in a stable order: H1, summary, details, H2 sections in the order you entered them. Read the Optional section, every note, and the labels to confirm they match the site before copying or downloading.
4. Copy or download the file. The download is named llms.txt. The text is deterministic, so re-running the same input produces the same output, which is useful when you want a reviewable artifact rather than an opaque crawl.
5. Validate the final edited file. Paste the edited draft into the validator to check for the required H1, heading order, link-list syntax, duplicate targets, and bounded size. The validator reports line-oriented issues and warnings and does not rewrite the file.
6. Publish it at the intended path. Serve it at /llms.txt, or at the appropriate subpath, with a plain-text or Markdown-compatible content type.
7. Periodically verify the referenced pages. Re-check that each canonical URL still resolves, that the content matches its note, and that no private or staging URL slipped in. Do not assume crawler adoption from publication alone.

What Belongs in a Useful llms.txt (and What Doesn't)

Curation matters more than length. A file that lists every page repeats the overload of a large sitemap and wastes a consumer's context window. Prefer canonical documentation, product explanations, policies, and stable reference pages. Link to Markdown versions when the site reliably serves them, but do not invent .md URLs that return errors.

Useful candidates to include:

• Top-level product or service pages that explain what the site offers.
• Documentation hubs and stable reference pages that AI assistants will reference when answering factual questions.
• Pricing, terms, and policy pages that frequently appear in user questions.
• Authoritative blog posts or research articles the site wants surfaced when relevant.

Things that usually do not belong:

• Authentication-gated pages, since the file has no way to grant access.
• Staging or internal URLs, since they leak preview content and break for outside readers.
• Every blog post on the site; pick the few that are canonical for your topic.
• Search result, filter, or pagination URLs, which add noise without adding signal.

The generator never scans a domain or sitemap. That is deliberate: a purely client-side form cannot prove which dynamic pages are canonical, current, accessible, or safe to recommend. The author remains responsible for selecting sources and checking each public URL.

llms.txt and the Other Discovery Files on Your Site

llms.txt is a complement, not a replacement, to the discovery files you may already maintain. The following comparison summarises the role each one plays, based on the documented scope of the llms.txt proposal and the standard purpose of each adjacent file:

Mechanism	Primary role	Format	Audience	Effect on access
llms.txt	Curated Markdown overview and short list of high-value references for inference-time use	Plain-text Markdown	Language models and AI assistants that choose to read it	None; does not block or grant access
robots.txt	Crawl preferences for participating crawlers	Plain-text directives	Web crawlers that follow the Robots Exclusion Protocol	Can disallow crawling of listed paths for compliant bots
sitemap.xml	Inventory of indexable URLs for search discovery	XML	Search engines that consume sitemaps	None; advisory list of URLs
Structured data (JSON-LD, Microdata)	Describes entities and page content for rich results	Embedded markup	Search engines and parsers that read structured data	None; describes existing content

Because the roles do not overlap, the practical guidance is to keep each file doing its own job. If you also need to tighten crawl rules, work through a practical robots.txt guide alongside this one rather than folding crawl directives into llms.txt.

Validating, Publishing, and Maintaining the File

After you download the file from the generator, review it as content rather than a mechanical SEO artifact. Confirm that the title and summary match the site, that every link is canonical, that notes describe real pages, that resources labelled optional are genuinely secondary, and that no private URL appears. Validation success on the draft means the file matches this tool's documented interpretation of the current proposal; it does not certify broader vendor support.

When you publish:

• Serve the file at /llms.txt, or at the appropriate subpath, with a plain-text or Markdown-compatible content type.
• Keep it small. The validator checks bounded size, and a long file defeats the curation point.
• Re-run validation whenever you edit the file by hand, since pasted edits can introduce duplicate URLs, broken list syntax, or misplaced headings.
• Periodically fetch each linked URL to confirm it still resolves and still matches its note.

The proposal and ecosystem can evolve, so recheck the methodology and source links before building automation that depends on exact syntax. Treat llms.txt as a maintained artifact: review quarterly, prune dead links, and refresh the summary when the site's positioning changes.

Setting Realistic Expectations

Because the format is an emerging proposal, it is worth being explicit about what publishing llms.txt does and does not do. It does not force any crawler or assistant to request the file. It cannot grant access to blocked pages, override authentication, remove content from model training, or prove that an AI answer will cite the site. The file can complement existing web controls, but it is not a substitute for them.

The honest way to measure value is to track whether your own tools and workflows use the file. If a documentation assistant, an internal research tool, or an in-house retrieval pipeline ingests /llms.txt, that is direct evidence the file is doing its job. Treat publication alone as evidence of nothing more than a published file. Crawler adoption and citation improvements, if they happen, will show up in your existing analytics and answer-tracking, not in the file itself.