What Is llms.txt? Format, Status and When to Use It

llms.txt is a proposed file that gives large language models a short, curated guide to a website: what the site is, and which pages matter most, with links to clean text versions of them. It's often described as "robots.txt for AI." That comparison is misleading. It doesn't control access at all. This guide explains what the proposal actually says, where it stands, and when creating one is worth your time.

Where it comes from#

The format is defined at llmstxt.org. Jeremy Howard first published it in September 2024. The problem it addresses is practical. Web pages are full of navigation, scripts, ads and layout markup, and a model or tool working with a limited context window gets far more out of a concise Markdown summary with links to the important pages than out of raw HTML.

It's a community proposal, not a standard from a standards body such as the IETF or W3C, and no one is required to support it.

The format#

An llms.txt file is Markdown placed at the root of a site (/llms.txt), or under a subpath such as /docs/llms.txt to cover that section. The proposal specifies this structure, in order:

1. An H1 with the name of the project or site. This is the only required section.
2. A blockquote with a short summary containing the key information needed to understand the rest of the file.
3. Zero or more Markdown sections (paragraphs, lists) with more detail about the project and how to interpret the files listed.
4. Zero or more H2 sections containing file lists: Markdown links, each optionally followed by a colon and a note about the page.
5. An "Optional" H2 section, by convention, for secondary links a tool can skip when it needs a shorter context.

An example file#

# NewBacklink

> NewBacklink offers free SEO and webmaster tools, including backlink,
> keyword, domain, technical SEO and AI visibility checks, plus guides on
> how to use them.

All tools are free and need no signup. Tool pages explain what each
check does and what its limits are.

## Tools

- [SEO Checker](https://newbacklink.com/tools/seo/seo-checker/): on-page and technical checks for a single URL
- [Robots.txt Checker](https://newbacklink.com/tools/technical-seo/robots-txt-checker/): test crawl rules for any user agent
- [AI Citation Checker](https://newbacklink.com/tools/ai/ai-citation-checker/): look up a brand in the AI Citation Index

## Guides

- [SEO audit checklist](https://newbacklink.com/blog/seo-audit-checklist/): step-by-step site audit
- [Allow or block AI crawlers](https://newbacklink.com/blog/how-to-allow-or-block-ai-crawlers/): robots.txt rules for AI agents

## Optional

- [Methodology](https://newbacklink.com/methodology/): how the AI Citation Index is collected

Notice what makes it useful: a plain-language summary, descriptive link notes, and a small number of carefully chosen pages. It isn't a dump of every URL on the site. Your sitemap already does that job.

Markdown versions of pages#

The proposal also suggests offering clean Markdown versions of important pages at the same URL with .md appended (for example page.html.md), or with the extension replaced (page.md). It mentions using rel="alternate" links with type="text/markdown" so tools can discover them. This is optional. It's most practical for documentation sites that already write content in Markdown.

What about llms-full.txt?#

You'll see llms-full.txt on many documentation sites. The name is a convention that grew up alongside the proposal, not a separate part of the core format. It usually holds the full text of the documentation in one Markdown file, so a user can paste or load the whole thing into a model at once. It makes sense for API and product documentation. For a marketing site, it rarely adds much.

The honest status#

This is where much of the online advice goes wrong, so be precise.

• Google says you don't need it. Google's AI features documentation states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." It adds that there are no additional requirements or special optimizations needed to appear in AI Overviews or AI Mode. For Google's AI features, normal SEO applies and llms.txt plays no documented role.
• No major AI assistant has published a commitment to use llms.txt when choosing which sites to cite, and there's no clear public evidence that having one increases citations or traffic. Be skeptical of any claim that it will.
• It does get used in tooling. Its clearest adoption is among developer documentation sites, and tools that let people feed documentation to a model. There, a concise index plus clean Markdown is genuinely convenient.
• It isn't access control. An llms.txt file doesn't allow or block any crawler. If you want to control AI crawlers, use robots.txt and server-level rules. See how to allow or block AI crawlers.

When it's worth creating one#

Good fit:

• Developer products, APIs and open-source projects whose users actively load docs into AI tools.
• Documentation sites that already have Markdown sources and can generate the file automatically.
• Sites with a large, messy structure where a curated index helps a person or tool find the canonical pages.

Low priority:

• Small business and marketing sites. Your time is better spent on clear page content, crawlability and third-party coverage.
• Anyone expecting a ranking or citation boost. There's no documented mechanism for one.

Creating a good file takes little effort, so if you fall into the first group, it's a reasonable small task. Just don't treat it as an AI visibility strategy.

How to create one#

1. Write the summary first. One or two sentences: what the site is, who it's for, and what it offers. This is the part a model is most likely to use.
2. Pick the pages that matter. Main product or tool pages, pricing, core documentation and key guides. Aim for quality over coverage.
3. Write a note for each link. Say what the page contains, not a marketing line.
4. Move secondary links under ## Optional.
5. Use absolute URLs pointing to canonical, indexable pages that return 200.
6. Serve it as plain text at /llms.txt and make sure robots.txt doesn't block it. You can test the path with the Robots.txt Checker.
7. Keep it current. A file that links to removed pages or old pricing does more harm than no file.

The LLMs.txt Generator can draft a file from your site, or check an existing /llms.txt against the llmstxt.org format and flag structural problems, such as a missing H1 or malformed link lists. Treat the generated draft as a starting point and edit the summary by hand.

Common mistakes#

• Listing hundreds of URLs, which turns the file into a second sitemap.
• Leaving out the summary blockquote, which is the most useful part.
• Linking to pages that redirect, return errors or carry noindex.
• Using it to state marketing claims that the linked pages don't support.
• Assuming it blocks or permits crawlers.

Where it fits#

Think of llms.txt as a courtesy file for tools that choose to read it. The factors that actually affect whether AI assistants describe and cite your brand are the same ones behind good SEO: clear content on your own pages, accessible pages, and credible coverage elsewhere. Our guide on how brands get cited by AI covers those in detail.