What Is llms.txt? Format, Example and SEO Reality

• The proposal was published by Jeremy Howard in September 2024.
• Only the H1 site or project name is required by the specification.
• A blockquote summary, explanatory notes and H2 link sections are optional.
• ## Optional has a specific purpose: its links can be omitted when a shorter context is needed.
• Google does not use the file for Search rankings, AI Overviews or other generative Search features.
• OpenAI and Anthropic document crawler controls in robots.txt, but neither documents llms.txt as a requirement for inclusion.
• The file can still be useful as a clean, maintained context map for agents, developer tools and manual AI workflows.

What is an llms.txt file?

An llms.txt file is a public Markdown document that describes a site and points to selected resources using human-readable links. The proposal addresses a real technical problem: ordinary HTML pages contain navigation, scripts, tracking, repeated templates and other material that consumes an AI system’s context window without helping it answer a question.

The file does not contain special AI metadata. It is plain text written in familiar Markdown syntax. A person can read it, an agent can fetch it, and a developer can use it as an input list when building retrieval or documentation workflows.

That distinction matters. llms.txt was proposed mainly for use at inference time, when a user or tool wants compact context about a site. It was not introduced as a universal crawling, training or ranking protocol.

What the specification actually requires

The official proposal keeps the format deliberately small:

1. An H1 with the site, project or organization name. This is the only required element.
2. An optional blockquote with a concise summary.
3. Optional Markdown paragraphs containing details that help a model interpret the links.
4. Zero or more H2 sections with Markdown link lists.
5. An optional description after each link, separated by a colon.

Here is a valid minimal example:

# Example Automotive Group

> Repair, rental and dealership services in the UAE.

Use the service pages for current offers and the case studies for evidence.

## Services

- [Car repair](https://example.com/car-repair/): Workshop services and booking details
- [Car rental](https://example.com/car-rental/): Fleet, locations and rental terms

## Proof

- [Case studies](https://example.com/case-studies/): Results with methods and dates

## Optional

- [Blog](https://example.com/blog/): Supporting guides and company updates

Use absolute URLs and factual descriptions. There is no official character limit or certification service. There is also no official rule that every site must have the file.

Why the Optional section matters

## Optional is not just another category name. The proposal gives it special meaning: an application creating a shorter context may skip those links. Put supporting articles, archives and secondary references there. Keep the core company definition, primary documentation, commercial pages and strongest proof in the main sections.

Does Google, ChatGPT or Claude use llms.txt?

The honest answer depends on what “use” means. A model or agent can fetch a public text file when instructed to do so. A documentation platform can expose it to developer tools. That does not mean a search engine uses it for discovery, ranking or citations.

The absence of documented support does not prove that no internal system ever fetches the file. It does mean there is no basis for promising clients that llms.txt will make a brand rank in ChatGPT, Claude or Google AI Overviews.

llms.txt vs robots.txt vs sitemap.xml

These files are often grouped together because they commonly sit near the domain root. They serve different jobs.

The priority order is straightforward. Fix crawl access, indexability, canonicalization, internal links and server-rendered content before spending time on llms.txt. A polished AI index cannot rescue pages that important crawlers cannot access.

What is llms-full.txt?

llms-full.txt is an ecosystem convention used by some documentation platforms to provide expanded page content in one large file. It is not the only companion format described around the original proposal. The official llms.txt project also discusses clean .md page versions and tools that generate context files such as llms-ctx.txt and llms-ctx-full.txt.

Most company sites do not need a full-text aggregate. It can become large, duplicate stale content and expose text that should not be distributed. It is especially inappropriate for gated or private documents. If critical information only exists behind client-side JavaScript, fix rendering and crawlability rather than using a public text dump as a workaround.

For a documentation site or API reference, a generated full-text file may be useful because developers often want a single context package. For a small service site, a concise index with strong destination pages is normally enough.

How to create an llms.txt file

1. Decide whether the site has a real use case

Create the file when the site has documentation, API references, a large knowledge base, several distinct service families or content that agents regularly need to navigate. For a five-page brochure site, it is optional. If the only goal is higher Google rankings, work on the pages themselves.

2. Choose canonical resources

Start with the pages that define the entity, explain the product or service, prove claims and answer recurring questions. Do not export the entire sitemap. A curated index should help a system choose the right source, not reproduce the navigation.

3. Write the H1 and summary

Name the organization or project consistently. The blockquote should state what the entity does, who it serves and, when relevant, its market or location. Avoid unverified superlatives such as “leading”, “best” or “most trusted”.

4. Group and annotate the links

Use sections that reflect real information needs: Documentation, Services, Definitions, Proof, Policies or Support. Describe what each page contains rather than repeating its title. Put non-essential resources under ## Optional.

5. Publish it at a stable URL

The common location is https://example.com/llms.txt. Serve a UTF-8 text response with a successful 200 status, no login wall and no redirect chain. Static sites can place the file in their public assets directory. A CMS can generate it, but automatic output still needs editorial review.

6. Validate the response and links

Check the file itself:

curl -I https://example.com/llms.txt
curl -s https://example.com/llms.txt

Then test every linked URL for redirects, errors, accidental noindex directives and inconsistent canonicals. Also confirm that the descriptions still match the visible page content.

7. Assign an owner and update trigger

Review llms.txt after migrations, URL changes, major product releases and documentation restructures. A quarterly check is a reasonable fallback, but event-based updates are more reliable. The file should be maintained with the content system, not launched once and forgotten.

How to add llms.txt in WordPress, Webflow and documentation platforms

The implementation method depends on where the website content lives:

• WordPress: Yoast SEO includes an llms.txt generator. Its technical specification says the plugin writes the file to the root directory, updates it weekly and can select pages automatically or manually. Review that selection rather than assuming the latest posts are the best sources.
• Webflow: Webflow lets site owners upload an llms.txt file through the SEO settings and serves it from the domain root. The platform also exposes llms.txt and Markdown pages for its own developer documentation.
• Mintlify: Mintlify automatically generates /llms.txt, /llms-full.txt and clean .md versions of documentation pages. This is a strong use case because coding assistants and AI tools often need compact technical documentation.
• Static sites and custom applications: Add a maintained file to the public assets or build output. A script can generate it from approved content records, but the final list still needs editorial control.

This platform support demonstrates publisher adoption, not automatic consumption by every large language model. A plugin can create the file; it cannot make an external AI system retrieve or trust it.

Who is actually using the proposed standard?

Adoption is clearest among documentation systems, CMS plugins and developer platforms. Webflow publishes an llms.txt endpoint for its developer documentation, Mintlify generates files for hosted docs, and Yoast provides generation for WordPress and Shopify. These are concrete examples of websites making machine-readable context available.

Consumption is harder to prove. Server logs can show a crawler or AI tool requesting the path, but they cannot reveal whether the information affected an answer. Published support by a content platform should never be rewritten as proof that ChatGPT, Claude, Perplexity or another AI-powered search product uses the file as a ranking signal.

What should go in llms.txt?

Include pages that help a model identify the entity and answer a user accurately:

• The canonical About or organization page
• Primary product, service or documentation pages
• Definitions and methodology pages
• Current pricing or plan information, when public
• Case studies with visible evidence and dates
• Editorial, review or data policies when trust matters
• Support and contact routes

Exclude private URLs, staging environments, gated downloads, account pages, internal search results, tracking parameters and thin archive pages. Do not place secrets or unpublished information in the file. It is a public resource that anyone can fetch.

For multilingual or versioned documentation, label the language and version explicitly. Separate sections or path-level files are easier to interpret than a mixed list of nearly identical URLs.

Does llms.txt help SEO or AEO?

llms.txt does not currently provide a confirmed SEO or AEO ranking benefit. Google has removed the ambiguity for its own products: adding the file neither helps nor harms visibility in Google Search or its generative AI features. No public evidence supports treating it as an AI citation shortcut.

It can still create operational value. Building the file forces a team to identify its canonical explanations, strongest proof and preferred entity language. Those decisions can improve the pages that search engines and retrieval systems actually use. The value comes from better information architecture and source content, not from the presence of the text file.

My priority stack for generative engine optimization is:

1. Crawlable, server-rendered and indexable source pages
2. Clear entity definitions and consistent naming
3. Direct answers backed by first-hand evidence and visible sources
4. Internal links that connect the full topic cluster
5. External authority and accurate third-party mentions
6. Structured data that matches visible content
7. llms.txt as an optional context and maintenance layer

The same principle applies when trying to rank in ChatGPT: build sources that deserve to be retrieved before adding experimental discovery files.

How to measure whether anyone uses it

Do not infer success from the file returning 200. Measure requests to the exact path in server or CDN logs. Record the user agent, request frequency, status code and whether the requester later accesses any linked pages.

Be careful with attribution. A request from an AI-related user agent proves only that the file was fetched. It does not prove that the content influenced an answer, citation or model training. Likewise, a ChatGPT referral does not prove that llms.txt caused the visit.

For a controlled test, give an internal agent permission to fetch the file, ask it questions that require different sections, and inspect which links it chooses. This evaluates whether the index is useful for an agent. It does not simulate an undocumented ranking system.

Common implementation mistakes

• Calling llms.txt an official ranking standard
• Copying every sitemap URL into the file
• Omitting descriptions or filling them with marketing claims
• Treating the file as permission for crawling
• Publishing private, gated or staging URLs
• Using relative links that break outside the site
• Generating a huge full-text file without a real retrieval use case
• Leaving redirects, 404 pages or old product names in the index
• Measuring fetches as if they were citations
• Spending more time on the file than on the pages it references

Should you create one?

Create llms.txt if it takes little maintenance and gives agents a genuinely better map of a complex public site. It is most defensible for documentation, APIs, knowledge bases and multi-service businesses. Keep it concise, factual and connected to canonical pages.

Skip or postpone it when crawlability, rendering, content quality or site architecture is still broken. Also skip it if nobody will own updates. The correct business case is “a cheap, machine-readable context index we can test”, not “a new file that will increase AI rankings”.

Primary sources

• The official llms.txt proposal and format
• Google Search guidance for generative AI features
• OpenAI crawler and robots.txt documentation
• Anthropic web crawler documentation
• Mintlify llms.txt and Markdown page implementation
• Yoast SEO llms.txt functional specification
• Webflow llms.txt publishing support