The llms.txt Verdict: 97% Unread. So Who Is It For?
The finding
In a study of 137,000 sites, 97% of valid llms.txt files received zero requests during May 2026. Of the small amount of traffic these files did attract, SEO audit tools were the largest single category of fetcher — larger than AI retrieval bots.
We have recommended llms.txt on this site. The evidence that arrived in 2026 says we were recommending it for the wrong reason. This post is the correction, and an argument for what the file is actually good for.
What llms.txt was supposed to do
The proposal is simple and appealing: publish a plain-text file at the root of your site listing your most important pages in a clean, model-friendly format, so an LLM does not have to reconstruct your information architecture from HTML soup. Think of it as a sitemap written for a reader rather than a crawler.
The format is a markdown file with an H1 title, a blockquote summary, and ##
sections containing lists of links with short descriptions.
Adoption grew fast. One count tracked roughly 4,000 files in June 2025 rising to over 36,000 by May 2026 — about a 9x increase, including a meaningful slice of the Fortune 500.
What actually happened
Adoption grew. Consumption did not.
The Ahrefs study is the clearest data point: across 137,000 sites, 97% of valid llms.txt files were never requested at all in the month studied. Where requests did occur, AI retrieval bots accounted for only about 1.1% of them, while SEO audit tools — software checking whether the file existed — made up over a fifth. A separate analysis across roughly 300,000 domains found no statistically significant relationship between having an llms.txt file and being cited in AI answers.
Then the providers spoke. In mid-2026 Google documented plainly that it does not use llms.txt for Search, for rankings, or for AI Overviews. OpenAI's documentation has never described consuming it. Anthropic publishes one and recommends the format in its guidance for writing documentation that agents can read — but does not consume it during crawling. Perplexity has no support for it.
There is a reasonable comparison to the meta keywords tag: a file that sites publish, that tools check for, and that the systems it is aimed at do not read.
Why we got it wrong, and why lots of people did
The mistake was a category error, and it is worth naming because it recurs constantly in this field.
llms.txt looks like robots.txt and sitemap.xml. Those files work because search engines committed to reading them and publishers could rely on that contract. llms.txt has the same shape but not the same contract — no major retrieval system agreed to fetch it. A standard is not a standard because it is well-designed and widely published; it is a standard because the systems on the other end read it.
The general lesson: when a technique's mechanism of action is "the AI will see this," ask which specific system reads it, and where that system's operator documented doing so. Applied honestly, that question disqualifies a large fraction of the advice currently sold as AI optimization.
So who is llms.txt for?
Here is the interesting part — the file did find a real constituency, just not the one it was pitched to.
Coding agents and developer documentation. When a developer points an agent at a library's documentation, an llms.txt file that enumerates the important pages is genuinely useful, because that agent is fetching URLs directly rather than relying on a search index. This is why several AI companies publish llms.txt for their own SDK documentation even though they do not consume anyone else's.
Agent-facing site structure. Google added llms.txt to Lighthouse's agentic-browsing audit category in 2026 — a telling reclassification. It moved out of the SEO bucket and into the "is this site usable by agents" bucket. That is the right home for it.
The distinction matters because it changes who should bother. If your site is developer documentation, an API reference, or anything an agent will be pointed at directly, publish one. If you are a marketing site hoping to appear in ChatGPT answers, it will do nothing for you, and the twenty minutes are better spent elsewhere.
Where llms.txt stands
- Publishing one is cheap and harmless. Nobody has found a downside.
- It will not improve your AI citation rate. No major answer engine reads it.
- It is genuinely useful for documentation sites that coding agents fetch directly.
- Follow the actual spec if you publish one — H1 title, blockquote summary,
##sections of markdown links. Many published files do not.
What to do instead
The techniques with evidence behind them are less novel and more boring, which is probably why the file got the attention.
Be retrievable at all. Server-rendered content, no crawler blocks on the search-side agents, stable URLs. Presence in the retrieval corpus is the precondition for every other tactic.
Be fresh. Cited URLs skew meaningfully fresher than the web at large. Real publication and modification dates, and content that has genuinely been revised.
Be quotable. Self-contained statements that survive extraction — a claim that depends on the previous three paragraphs cannot be lifted into an answer.
Be attributed. Named sources with resolvable links. Answer engines are staking their credibility on the claims they repeat, which gives them reason to prefer sources that show their work.
Be present where the engines look. Reddit is the most-cited domain across multiple engines, accounting for roughly 16.7% of ChatGPT citations in one 2026 analysis. Your owned site is not the only surface that matters.
None of that is as satisfying as adding a file. It is what the evidence supports.
References
- Ahrefs, "We studied 137,000 sites. Almost nobody is reading your llms.txt", June 2026
- Google Search Central, AI features and your website
- Ahrefs, "Most cited domains in ChatGPT", July 2026
- llmstxt.org — the specification
Start with what works
Our quick start covers the changes with actual evidence behind them, in priority order.