GEO

What Is llms.txt, and Do You Need One?

ReadRankAI Team · November 3, 2026 · 9 min read

A plain text llms.txt file with Markdown headings and links, connected to an AI node, illustrating what llms.txt is

The short answer: What is llms.txt? It’s a plain text file in Markdown, placed at yoursite.com/llms.txt, that gives AI tools a short summary of your site and a list of your most important pages. It was proposed in 2024 as a way to help large language models find the right content quickly. But in 2026, Google says you don’t need it for its AI search features, and server-log data shows AI search bots rarely request it. For most websites it’s optional. It’s most useful for documentation and API sites that coding assistants read.

So should you make one? Probably not as a priority. This guide explains what llms.txt is, what the file looks like, what the latest data says about who actually reads it, and what to do instead if your goal is to show up in ChatGPT, Perplexity and Google’s AI answers.

Key Takeaways

llms.txt is a guide, not a gate. It suggests pages to AI tools. It doesn’t allow or block anything. Access is still controlled by robots.txt.
Google doesn’t use it for search. Google’s May 2026 guide to its AI search features says you don’t need llms.txt or Markdown versions of your pages.
Almost nobody requests it yet. In Ahrefs’ study of 137,210 sites, 97% of llms.txt files got zero requests in May 2026. AI retrieval bots made just 1.1% of the requests.
It helps one group. Documentation and API sites, like Anthropic, Stripe and Cloudflare, publish llms.txt files for coding assistants and developer tools.
robots.txt matters far more. If your robots.txt blocks AI search crawlers, no llms.txt file can fix it.

What Is llms.txt?

llms.txt is a proposal by Jeremy Howard, first published in September 2024. The idea is simple: websites are built for people, with menus, ads and scripts that make them hard for AI tools to read in a short time. A single Markdown file at the root of your site could give an AI tool the key facts and point it to the pages that matter, in a format it reads easily.

The name copies robots.txt, but the two files do very different jobs:

	robots.txt	llms.txt
What it does	Tells crawlers what they may and may not visit	Suggests which pages AI tools should read
Can it block a bot?	Yes	No
Who follows it	Major search engines and most AI crawlers	No major AI search product has said it uses it
Effect on rankings	Indirect: blocked pages can’t be crawled	None that anyone has shown

The proposal also suggests offering clean Markdown versions of important pages at the same address with .md added, so an AI tool can read the text without the page layout.

What an llms.txt File Looks Like

The format is plain Markdown. According to the official specification, only the H1 with your site or project name is required. Everything else is optional, in this order:

An H1 with the name of your site or project.
A short summary in a blockquote (a line starting with >).
Extra details in normal paragraphs or lists.
H2 sections with lists of links, each written as [name](url), with an optional note after a colon.
An “Optional” section for links an AI tool can skip when it needs a shorter version.

Here’s an example for a made-up analytics company:

# Acme Analytics

> Acme Analytics is web analytics software for small online stores. It tracks visitors, sales and ad results in one dashboard.

Acme works with Shopify, WooCommerce and custom sites. Plans start at $19 a month.

## Docs

- [Quick start](https://acme.example/docs/quick-start.md): install the tracking code in 5 minutes
- [Reports](https://acme.example/docs/reports.md): what each dashboard report shows
- [API reference](https://acme.example/docs/api.md): endpoints, limits and authentication

## Company

- [Pricing](https://acme.example/pricing.md): plans and limits
- [Contact](https://acme.example/contact.md): support hours and email

## Optional

- [Changelog](https://acme.example/changelog.md): release notes

You may also see a file called llms-full.txt. It isn’t in the llmstxt.org specification. It’s a common extra where a site puts the full text of its docs into one big file. Anthropic, for example, publishes a short llms.txt index plus a much larger llms-full.txt.

Does Anyone Actually Use llms.txt?

This is the question that matters, and the evidence in 2026 is clear.

Google Search: no

Google Search has been consistent. In June 2025, Google’s John Mueller wrote that “no AI system currently uses llms.txt”. At a Google event, Gary Illyes was reported saying Google doesn’t support llms.txt and isn’t planning to. In June 2026, Mueller called it “purely speculative for now”. He has also compared it to the old keywords meta tag.

Most important, Google’s own guide to optimizing for its generative AI features, published in May 2026, lists llms.txt among the things you don’t need for AI Overviews and AI Mode, along with Markdown copies of your pages and content chunking.

AI crawlers: almost never

In June 2026, Ahrefs analyzed server logs from 137,210 websites:

28% of the sites had an llms.txt file. Ahrefs calls this an upper bound, because its users are more technical than average.
97% of those files got zero requests in May 2026. The other 3% received all the llms.txt traffic.
SEO audit tools made the most requests (21.7%), followed by unidentified bots (14.9%) and general web crawlers (13.1%).
AI retrieval bots, the ones that fetch pages to answer questions in AI search, made only 1.1% of the requests. OpenAI’s training crawler GPTBot made about 4.5%.

Ahrefs’ conclusion: for AI search visibility, “the cons outweigh the pros right now.”

Browser agents and developer tools: a little

There are two places where llms.txt does get some attention:

Coding assistants. In the same Ahrefs data, Claude Code fetched llms.txt files more often than every AI retrieval bot, assistant and training crawler except GPTBot and one indexing bot. It’s also why many developer platforms, like Stripe and Cloudflare, publish one.
Chrome’s Lighthouse. In May 2026, Lighthouse 13.3 added an llms.txt check in a new Agentic Browsing category. It’s about how browser agents understand a site, not about rankings or AI citations. If you don’t have the file, Lighthouse marks the check as not applicable instead of failing it.

So Google’s teams don’t fully agree. Google Search says you don’t need it, while a Chrome tool checks for it. For search visibility, the Search team’s guidance is the one that counts.

Do You Need an llms.txt File?

It depends on what kind of site you run:

Your site	Do you need llms.txt?
Developer docs, an API or a software product with documentation	Worth adding. Coding assistants and developer tools do fetch it, and it’s cheap to make.
A blog, a local business or an online store	Optional. It won’t improve your Google rankings or AI citations based on current evidence.
Your platform creates one automatically	Leave it on. It costs nothing, but check that it lists your real key pages.

What it won’t do: there’s no evidence it gets you into ChatGPT answers, it can’t block AI training, and it doesn’t replace a working robots.txt.

How to Create an llms.txt File

If you decide to add one, keep it small and simple:

Write the H1 and summary. Your site name, then one or two sentences on what you do and who it’s for.
List your key pages. Group 5 to 20 important pages under H2 headings, each with a one-line note. Skip tag pages, login pages and thin pages.
Use plain links only. Ahrefs recommends treating the file like code: keep it to plain links and limit who can edit it, because text that AI agents read can carry hidden instructions.
Upload it to your root folder, so it opens at yoursite.com/llms.txt, and check it loads as plain text.
Link to it from your docs or footer if you want agents to find it. Ahrefs found agents fetch it mostly when they’re pointed to it.

Keep it updated when you add or remove important pages. An llms.txt full of broken links helps no one.

What Matters More Than llms.txt for AI Visibility

If your real goal is to show up in AI answers, these matter far more:

Let AI search crawlers in. Make sure your robots.txt doesn’t block OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot or Googlebot. You can check all of them in a few seconds with our free AI Crawler Checker. It also shows whether you have an llms.txt file.
Answer the question first. Pages that put a clear answer in the opening lines are easier for AI tools to quote. Our guide on how to get cited by ChatGPT covers the details.
Rank in normal search. Google’s AI Overviews draw on pages that already rank, as we explain in how to get cited in Google AI Overviews.
Get mentioned on trusted sites. AI tools lean heavily on what Reddit, YouTube and LinkedIn say about you. See how to get mentioned by AI.

FAQ

Does llms.txt help SEO?

No, based on current evidence. Google says it doesn’t use llms.txt for Search or its AI features, and no published study we know of shows it improves rankings or AI citations.

Does ChatGPT read llms.txt?

OpenAI hasn’t said ChatGPT uses llms.txt to answer questions. In Ahrefs’ data, OpenAI’s training crawler GPTBot requested llms.txt files sometimes, but AI retrieval bots, the ones that fetch pages for answers, made only 1.1% of all requests.

Can llms.txt block AI crawlers or AI training?

No. llms.txt only suggests pages. To block or allow a crawler, use robots.txt, for example by disallowing GPTBot to opt out of OpenAI’s model training while still allowing OAI-SearchBot for ChatGPT search.

What’s the difference between llms.txt and llms-full.txt?

llms.txt is a short index with links. llms-full.txt is a common extra, not part of the original proposal, that puts the full text of a site’s docs into one file. Developer documentation sites often publish both.

Where do I put the llms.txt file?

In the root folder of your website, so it opens at yoursite.com/llms.txt. The proposal also allows a file in a subfolder, such as /docs/llms.txt, to cover just that section.

The Bottom Line

llms.txt is a reasonable idea that hasn’t caught on with the tools that matter for search. Google says you don’t need it, and AI search bots rarely request it. If you run developer docs, add one, because coding assistants do fetch it. For everyone else, it’s a 15-minute extra, not a strategy.

Spend your time on what AI search actually reads: an open robots.txt for AI search crawlers, clear answers on your pages, strong rankings and real mentions on the sites AI tools trust.