llms.txt: what it is and whether you need it

llms.txt is a proposed plain-text file, written in Markdown and placed at the root of a website, that gives language models a short summary of the site and links to its best pages. It is cheap to add and harmless, but no major AI search product has confirmed that it reads the file, so do it after the basics, not instead of them.

Where did llms.txt come from?

Jeremy Howard of Answer.AI proposed it in September 2024. The reasoning is that language models have limited context windows, and a full website, with its menus, scripts and clutter, is hard for them to read. A clean, curated file could point a model at the right pages in a form it can use directly.

The idea sits alongside two older files. robots.txt tells crawlers where they may go. sitemap.xml lists URLs for search engines. llms.txt is meant to be a reading guide for AI.

What does an llms.txt file contain?

The proposal sets a simple format in Markdown:

1. A title in an H1 heading with the name of the site or project.
2. A short summary in a blockquote.
3. Optional paragraphs with extra context.
4. Sections under H2 headings that hold lists of links, each with a brief note.
5. An optional section for lower-priority links.

A small example for a service business:

# Example Studio

> Example Studio is a web and software studio in Pune, India, working with small and mid-sized businesses on websites, applications and search visibility.

## Services

- [Web design and development](https://www.example.com/services/web): Business websites, e-commerce and site care.
- [Search visibility](https://www.example.com/services/search): Technical SEO, local SEO and AI answer visibility.

## Guides

- [Local SEO for a Pune business](https://www.example.com/insights/local-seo): How local search works and what to fix first.

## Company

- [About](https://www.example.com/about): Who we are and how we work.
- [Contact](https://www.example.com/contact): How to reach us.

A companion file, llms-full.txt, is sometimes published with the full text of key pages in one document, usually for documentation.

Does anyone actually use it?

Here the picture is thin. The file has been adopted by a number of documentation sites and developer tools, where coding assistants and agents can fetch it to understand an API or library. That is a real use.

For search and answer products, there is no published confirmation. Google has not said Search or AI Overviews use it, and Google's John Mueller has said publicly that he is not aware of any AI service using it and compared it to the old keywords meta tag. OpenAI, Anthropic, Perplexity and others have not announced that they rely on it for citing websites. Server logs on some sites show occasional fetches by AI-related bots, which is not proof of influence.

Things may change, and some tools may use it now without telling anyone. But right now it is a hopeful standard, not a lever.

Should you add one?

Use this rule of thumb.

Add it if:

• You publish documentation, an API, a product with technical guides, or a large body of reference content. Developer tools and coding assistants can use it.
• You already have clean content and the file takes under an hour to write.
• You want a tidy, current summary of your site in one place, as a discipline.

Skip it for now if:

• Your site still has crawl, speed or content problems. Fix those first.
• You would need to keep a long file in sync by hand and have no process for it.
• Someone is selling it to you as a way to get into ChatGPT. It will not do that on its own.

For most small service businesses, it is optional and low priority.

What matters more for AI visibility?

The things that Google and the AI vendors do document:

1. Crawlers can reach your pages. Check robots.txt, firewalls and bot protection.
2. Pages are indexed and render without heavy JavaScript.
3. Each key page opens with a direct answer.
4. Facts about your business are clear, consistent and backed by structured data.
5. Trusted third parties mention and review you.

The steps are set out in how to show up in ChatGPT and Google AI Overviews. Terminology is covered in AEO vs SEO vs GEO.

How do you create and maintain one?

1. Decide what a model should know about you in five lines.
2. List the 10 to 30 pages that explain your business best.
3. Write each link with a one-line plain description.
4. Save it as llms.txt at the root, so it loads at yourdomain.com/llms.txt with a plain-text or Markdown content type.
5. Link only to public pages.
6. Update it when you add or retire key pages. If your site is built from a content system, generate the file from the same data as your sitemap so it stays current.
7. Optionally, confirm it in your server logs and see which bots fetch it.

What are the trade-offs?

The cost is small: an hour to write and a little upkeep. The risk is that it goes stale and points to dead pages. Another risk is distraction, with time spent on an unproven file while real problems remain.

There is also an expectation risk. If you tell your team that the file will get you cited, you will be disappointed. Treat it as housekeeping.

Where does TechSum fit?

We include an llms.txt in the sites we build, generated from the site's content, as one small part of AI readiness. The larger work is crawler access, answer-first content and structured data. See the AEO and GEO page and the Search and Answer Visibility pillar. The free health check tests whether AI crawlers can read your site.