An llms.txt file is a short, hand-curated Markdown document that lists a site's most useful resources for language-model readers, written to the structure proposed at llmstxt.org. Generating one for your own website does not require crawling, scraping, or sending anything to an external service: you decide which canonical URLs belong on the index, paste the editorial inputs into a local form, and the tool emits deterministic Markdown in the proposal's documented order. That single-file, curator-driven approach is what separates llms.txt from a sitemap or a robots directive, and it is also why a generator that never touches your server tends to produce more honest results than an automatic scanner, since only you can confirm which pages are stable, canonical, and safe to recommend. The workflow below walks through the editorial choices you make before generation, the step-by-step operation of the generator, the validation pass that confirms the draft follows the proposal, and the publishing and verification loop that keeps the deployed file useful over time.
What llms.txt Actually Is (and Isn't)
The proposal describes a small Markdown index with one required element: a single H1 containing the project or site name. After that H1, the proposal allows an optional blockquote summary, optional non-heading details, and zero or more H2 sections, each holding Markdown list items whose required core is a link. The Optional section has its own convention in the proposal: resources placed under an H2 named Optional are flagged as secondary material that a model can skip when it needs a shorter context. Everything else, including which resources qualify as optional, is an editorial decision the file's author makes, not a label the generator assigns automatically.
Equally important is what llms.txt does not do. It is not a robots directive, not a security control, not a sitemap replacement, and not a guaranteed discovery protocol. Publishing the file does not force a crawler or an assistant to request it, cannot grant access to blocked pages, cannot override authentication, and cannot remove content from model training. It is also not a ranking signal on its own: there is no evidence that a search engine treats the presence or absence of llms.txt as a ranking factor, and the proposal makes no such claim. Treating it as one tends to produce files stuffed with every page, which defeats the purpose of curation.
| Mechanism | Primary role | Format | Consumed by |
|---|---|---|---|
| llms.txt | Curated inference-time resource map | Markdown | Language models and assistants |
| robots.txt | Crawl preferences for participating crawlers | Plain text | Web crawlers that opt to read it |
| sitemap.xml | Inventory of indexable URLs | XML | Search engines |
| Structured data | Description of entities and page content | JSON-LD or Microdata | Search and assistant parsers |
Each mechanism keeps its own purpose. The llms.txt Generator handles the first row of that table only, and it does so without crawling your site or replacing any of the others.
Curating Canonical Links Before You Generate
Before opening the generator, the most important work is editorial. Curation matters more than length, and the proposal is explicit that a file containing every page can repeat the overload of a large sitemap and waste a model's context. The right starting set is small, factual, and stable: canonical documentation, product explanations, policies, and reference pages that return their intended content today and are likely to keep doing so tomorrow.
Three checks belong in this pre-flight phase. First, confirm every URL is canonical: the version you actually want a model to read, not a staging, draft, or duplicated path. Second, prefer Markdown versions when the site reliably serves them, but do not invent .md URLs that return errors, because a 404 on the list is worse than a missing entry. Third, decide which resources are truly secondary, since labelling too much as Optional reduces the file's usefulness while labelling too little leaves a short-context reader overwhelmed. Any URL containing credentials or pointing at executable or embedded-data schemes belongs on the floor, not in the form; the generator will reject them, but skipping them upstream keeps the draft review shorter.
How to Generate llms.txt From a Website Locally
The generator runs entirely in the browser. The following sequence is what an operator can expect from start to finish.
- Open the llms.txt Generator and enter the site or project name. That string becomes the required H1.
- Add an optional blockquote summary that captures the site in one or two sentences, then any free-form details you want readers to see before the link sections. Leave either blank if you do not want them.
- Create H2 section headings and add only the canonical links each section should contain. Each list item needs a link at its core; you can follow the link with a colon and a short note explaining what the resource contains.
- Decide which section, if any, should be named Optional. Move the genuinely secondary resources into it, leaving the most useful links above.
- Generate the Markdown. The output is deterministic plain text in the proposal's order: H1, summary if present, details if present, then H2 link sections.
- Review the Optional section and every note. Read the file as content rather than as a mechanical artifact, and confirm the title and summary still match the site.
- Copy the result to your clipboard or download it as llms.txt. The download filename is fixed; choose a different path on your end if you need a different name.
- Paste the edited file back into validation mode before publishing. Confirm the validator reports no unaddressed errors.
- Publish the final file at /llms.txt or the appropriate subpath, served as plain text or Markdown, then schedule a periodic check of every listed URL.
Validating the Draft Against the Proposal's Order
Validation is a separate mode from generation, and it is worth treating it as one. Pasted drafts are checked for the required H1, the heading order from the proposal, link-list syntax, duplicate targets, and bounded size. The validator reports concrete, line-oriented issues and warnings, and it does not rewrite the file behind your back. It also does not fetch any listed URL or claim that the linked resource is accurate, which keeps the responsibility for source verification where it belongs.
Two warning categories deserve a second look. Duplicate normalized URLs are reported rather than silently multiplied, because a list that lists the same target twice tells a model the same page matters twice. Unsafe URLs, including any with embedded credentials or with executable or embedded-data schemes, are rejected on serialization, so catching them in validation means the draft came from outside the generator and needs the same scrutiny. Validation success means the draft matches the generator's documented interpretation of the current proposal; it is not a certification of broader vendor support, and the proposal and ecosystem can evolve. Recheck the methodology and source links before you build any automation that depends on exact syntax.
Publishing and Verifying Over Time
Publishing is the moment the file starts to represent the site, so two details matter. The path should be /llms.txt or a clearly equivalent subpath, and the content type should be plain text or Markdown-compatible so consumers can fetch it without negotiation surprises. Once the file is live, measure whether your own tools and workflows actually use it, and do not treat publication alone as evidence of search or citation improvement.
Verification is the part most fileskip. A reference list drifts when pages move, get renamed, or fall behind authentication. Schedule a periodic check of every linked URL, replace any that no longer return the intended content, and trim links that have become non-canonical. The proposal remains emerging, and the publisher's job is to keep the curated map honest rather than to wait for ecosystem support to confirm it. If you ever need a deeper walkthrough of the proposal's section ordering, the guide on generating llms.txt for a website in the right order pairs naturally with this workflow, and the DeveloperHub implementation documentation and the AnswerDotAI reference repository are useful next reads when you want to see how others have structured their own files.
A llms.txt file is small on purpose, and the value of the llms.txt Generator is the discipline it enforces: one required H1, a deterministic Markdown order, an explicit Optional section, and a validation pass that catches the mistakes a hand-typed draft tends to make. Combine those mechanics with editorial curation and a recurring link check, and you have a file that does what the proposal actually promises: a concise, reviewable Markdown overview of the site's most useful resources, with no claim of crawler adoption or ranking gains attached.