Generating an llms.txt file means producing a small Markdown index with one required H1 title, an optional blockquote summary, optional explanatory details, and zero or more H2 sections that group canonical links by purpose. The file follows the format proposed at llmstxt.org, and its structure is deliberately predictable so both language models and ordinary parsers can read it without ambiguity. Order matters: the H1 comes first, the summary follows when present, free-form details appear before any H2 sections, and the H2 sections themselves have a defined position. Curation matters more than length. A file that links to every page on a site repeats the overload of a large sitemap and wastes a consumer's context; a short, deliberately chosen list of canonical documentation, product explanations, and stable reference pages serves better. llms.txt is an emerging proposal, not a robots directive, security control, sitemap replacement, or guaranteed discovery protocol, so generation is a starting point, not a finish line. The job is to make the curation explicit, the Markdown deterministic, and the editorial decisions reviewable.
What Goes Into an llms.txt File
The proposal fixes a strict order so the same file reads the same way across parsers. The required element is a single H1 containing the project or site name. After that you may add a blockquote summary, additional non-heading details, and zero or more H2 sections. Each H2 section contains Markdown list items whose required core is a link, and an optional colon plus note can describe what the linked resource covers. Labels, notes, headings, and summaries are normalized so control characters or accidental Markdown delimiters cannot break the structure, and URLs are reviewed before serialization: HTTP and HTTPS are accepted, while credential-bearing schemes and executable or embedded-data schemes are rejected. Duplicate normalized URLs are reported instead of silently multiplying entries.
The proposal also defines a convention for an Optional section. Links under an H2 named Optional identify secondary material that can be skipped when a shorter context is needed. The tool does not automatically label a resource optional — that is an editorial decision about which sources are essential for understanding the site.
| Section | Required? | What it carries |
|---|---|---|
| H1 with site name | Yes | Identifies the document and the project it describes |
| Blockquote summary | No | One-paragraph overview of the site's purpose |
| Free-form details | No | Context placed before any H2 link sections |
| H2 link sections | No (zero or more) | Grouped canonical resources by topic |
| Optional H2 section | No | Secondary material skippable in shorter contexts |
Why a Local-Only Generator Changes the Workflow
When the generator runs entirely in your browser, the inputs never leave the page. The llms.txt Generator creates a small Markdown index that follows the format proposed at llmstxt.org, but it does not crawl your domain or send your URLs to an external service. That separation matters because no purely client-side form can prove which dynamic pages are canonical, current, accessible, or safe to recommend. You remain responsible for choosing the resources, and the tool's job is to serialize your decisions into a predictable document and to validate any pasted draft against the same documented interpretation.
The deterministic output also means two runs with the same inputs produce identical text. That property lets you regenerate the file after edits without drift between teammates, and lets the validator treat the file as content rather than a magical artifact. The output is plain text, the download is named llms.txt, and the same Markdown structure is what you publish at the intended path.
How to Generate an llms.txt File in Three Passes
The workflow is a short loop: assemble your inputs, generate and review, then validate and publish. The steps below cover what the tool actually controls and what you still own as the editor.
- Enter the site or project name as the required H1. This is the only mandatory input; everything else is optional.
- Add a blockquote summary if you want a one-paragraph overview, then free-form details that explain the project before any sections appear.
- Group your canonical high-value links under clear H2 headings. Prefer documentation, product explanations, policies, and stable reference pages over a full site dump.
- Generate the deterministic Markdown, read the Optional section and every note, then copy the file or download it as llms.txt.
- Validate the final edited file. Paste the draft into the validator on the same generator page to check for the required H1, heading order, link-list syntax, duplicate targets, and bounded size.
- Publish llms.txt at /llms.txt or the appropriate subpath with a plain-text or Markdown-compatible content type.
- Re-verify the referenced pages periodically and re-run the validator after edits. Validation success means the draft matches the tool's interpretation of the proposal; it does not certify broader vendor support, and the ecosystem can evolve.
For readers who want the validation pass treated as its own review step, the guide on how to generate llms.txt and validate it before publishing walks through the same two-pass flow with deeper examples.
Editing the Generated Markdown Before Publishing
Downloading the file is not the end of the workflow — it is the beginning of editorial review. Open the generated llms.txt as content, not as a mechanical SEO artifact. Confirm that the H1 and blockquote summary match the site as you would describe it to a new colleague. Walk through every link and ask whether each URL is canonical: does it return the intended content today, or does it redirect to a generic landing page, a marketing variant, or a deprecated path?
Review the notes. The tool lets you append an optional colon and explanation after each link; that note must be factual and concise. Avoid stuffing notes with marketing language or duplicating the link label. Check the Optional section. The tool does not automatically label a resource optional — that is an editorial decision about which sources are essential for understanding the site. If everything under an Optional heading is actually critical, rename the heading. Conversely, if a "core" section holds secondary material, move it under Optional.
Finally, scan for private URLs, staging paths, or credentials that may have slipped into a note. The generator rejects credential-bearing schemes, but a copy-pasted note can still leak internal hosts, and a stray hash symbol can break the heading order if the file is edited outside the tool. Run the validator after any hand-edit before publishing.
Deploying and Re-Verifying the File
After the file passes review, publish it at the intended path. Most teams publish at /llms.txt with a plain-text or Markdown-compatible content type. Subpaths are also valid when the curated overview describes a specific subdirectory, such as /docs/llms.txt for documentation portals. The file should sit alongside existing controls rather than replace them. The DeveloperHub llms.txt documentation describes how robots.txt expresses crawl preferences for participating crawlers, sitemap.xml inventories indexable URLs for search discovery, structured data describes entities and page content, and llms.txt offers a curated Markdown overview for inference-time use. Each mechanism keeps its own purpose.
Re-verification is part of the contract. AnswerDotAI's llms-txt reference repository is one of the public fixtures worth checking when the proposal changes. Periodically re-run the validator after editing the deployed file, and re-test the published URLs by hand. llms.txt is an emerging proposal, not a robots directive, security control, sitemap replacement, or guaranteed discovery protocol. Publishing the file does not force a crawler or assistant to request it. It cannot grant access to blocked pages, override authentication, remove content from model training, or prove that an AI answer will cite the site. Measure whether your own tools and workflows use the file; treat publication as a starting point rather than evidence of search or citation improvement.
Pitfalls That Break a Reviewed llms.txt
A short list of mistakes catches most failed files:
- Inventing .md URLs that do not exist. Only link to Markdown versions when the site reliably serves them; otherwise link to the HTML canonical.
- Linking every page on the site. A long index repeats the overload of a big sitemap and wastes consumer context.
- Treating llms.txt as a ranking file. It is a curated Markdown overview, and adoption is voluntary.
- Editing outside the generator and then publishing without re-running the validator. Markdown is fragile; a stray hash in a note can break heading order.
- Calling everything Optional. Optional means secondary; mislabeling dilutes the meaning of the section.
If you're weighing options, Default robots.txt for WordPress: A Safe Starting Point covers this in detail.