The finding
Ahrefs studied llms.txt (a plain-text file meant to point AI models to a site's most important pages) across 137,000 sites. During May 2026, 97% of the valid files received zero requests. Of the little traffic these files did get, SEO audit tools (software that checks a site for search problems) were the largest single group of visitors, ahead of the bots AI companies use to fetch pages.
We have recommended llms.txt on this site. The evidence that arrived in 2026 says we were recommending it for the wrong reason. This post is our correction, and an argument for what the file is actually good for.
What llms.txt was supposed to do
The idea is simple and appealing. You put a plain-text file called llms.txt at the top level of your site, listing your most important pages in a clean format that AI models find easy to read. Then an LLM (large language model, the kind of AI behind ChatGPT and Claude) doesn't have to work out how your site fits together from messy page code. It's meant to be a sitemap (the list of pages a site publishes for search engines) written for a reader, where the usual sitemap is written for a crawler (the program a search engine sends to fetch pages).
The format is a markdown file (plain text with a few symbols for formatting) with a title, a short summary, and sections listing links with a line of description each.
Plenty of sites took it up. One count found roughly 4,000 llms.txt files in June 2025 and more than 36,000 by May 2026, about nine times as many, including a good number of Fortune 500 companies.
What actually happened
Almost nothing reads the files sites have published, and the Ahrefs study is the clearest evidence. Across 137,000 sites, 97% of valid llms.txt files weren't requested even once in the month studied. Where requests did happen, the bots AI companies use to fetch pages made only about 1.1% of them, while SEO audit tools checking whether the file existed made more than a fifth.
Google, for one, has been clear. In mid-2026 it stated plainly in its documentation that it doesn't use llms.txt for Search, for rankings, or for AI Overviews. We couldn't find anything from OpenAI saying it reads the file. Anthropic publishes an llms.txt of its own, but we found nothing saying its systems read other sites' files when they fetch the web, and the same goes for Perplexity.
A separate analysis of roughly 300,000 domains found no meaningful link between having an llms.txt file and being cited in AI answers.
The closest comparison is the old meta keywords tag (a hidden keyword list search engines stopped reading years ago): something sites publish, that tools check for, and that the systems it's aimed at ignore.
Why we got it wrong, along with lots of other people
We mistook what kind of thing the file was. It's an easy mistake, and it comes up again and again in this field, so it helps to spell it out.
llms.txt looks like robots.txt (the file that tells search engine bots which pages they may visit) and sitemap.xml. Those files work because search engines committed to reading them, and publishers could count on that promise. llms.txt has the same shape but no such promise, because no major AI search system agreed to fetch it. A well-designed, widely published file still isn't a standard until the systems on the other end read it.
So when a technique works on the theory that "the AI will see this," ask which specific system reads it, and where that system's owner says so in writing. Asked honestly, that question rules out a large share of the advice currently sold as AI optimization.
So who is llms.txt for?
The file did find an audience, though a different one from the people it was pitched to.
Coding agents and developer documentation. When a developer points a coding agent (an AI assistant that writes and fixes code) at a library's documentation, an llms.txt file listing the important pages is really useful. That agent fetches pages directly instead of relying on a search index. That's why several AI companies publish llms.txt for the documentation of their own developer tools, even though their systems don't read anyone else's.
Telling agents how a site is organized. In 2026 Google added an llms.txt check to Lighthouse, its free tool for auditing web pages, as one of its agentic browsing checks (tests of how well an AI agent can find its way around a page and use it). That moved the file out of the SEO bucket and into the "can agents use this site?" bucket, which is the right home for it.
This changes who should bother. If your site is developer documentation, the reference for an API (a set way for other software to ask your service to do things), or anything else an agent will be pointed at directly, publish one. If you run a marketing site and hope to appear in ChatGPT answers, it will do nothing for you, and your twenty minutes are better spent elsewhere.
Where llms.txt stands
- Publishing one is cheap and harmless. Nobody has found a downside.
- It won't raise how often AI answers cite you. No major answer engine reads it.
- It's useful for documentation sites that coding agents fetch directly.
- If you publish one, follow the specification: a title, a short summary, then sections of links in markdown. Many published files don't.
What to do instead
The techniques with evidence behind them are older and more boring, which is probably why a new file got so much attention.
Make sure AI tools can fetch your pages at all. Send the content in the page your server delivers, don't block the bots that AI search tools use, and keep your page addresses stable. Nothing else works until your pages are among the ones these tools can pull from.
Keep your content fresh. In a July 2025 study of about 17 million citations, Ahrefs found that pages cited by AI assistants were about 25.7% newer on average than the pages in Google's ordinary search results (about 1,064 days old against 1,432), though Google's AI Overviews tended to cite older pages. Show real publication and update dates, and only change the date when you've actually revised the content.
Write sentences that can be quoted on their own. A claim that only makes sense after the previous three paragraphs can't be lifted into an answer.
Show your sources. Name them and link to them. Answer engines stake their credibility on the claims they repeat, which gives them good reason to prefer sources that show their work.
Be present where the engines look. In July 2026, Ahrefs' list of the domains ChatGPT cites most put Reddit first, with 16.7% of citations. By November, Ahrefs had Reddit at 6.2%, behind Wikipedia, but it was still near the top. Your own site isn't the only place you can show up.
References
- Ahrefs, "We studied 137,000 sites. Almost nobody is reading your llms.txt", June 2026
- Google Search Central, AI features and your website
- Ahrefs, "New Study: AI Assistants Prefer to Cite “Fresher” Content", July 28, 2025
- Chrome for Developers, llms.txt (Lighthouse agentic browsing audit)
- Ahrefs, "Most cited domains in ChatGPT", July 2026
- llmstxt.org, the specification