Everyone Writes llms.txt. Almost No AI Reads It.

Recently we ran our own site through PageSpeed Insights (PSI) and met its newest category: "Agentic Browsing." We scored 2/2, and llms.txt — the file everyone keeps recommending lately — sat quietly under "not applicable." Curious, we pulled the industry data, and the contrast is almost comic: in seven months, llms.txt went from 0.4% of websites to one in ten1; this May, 97% of llms.txt files were not fetched a single time2.

The whole web is writing it; almost nothing is reading it. Here is what the file is, how it got hot, who actually reads it — and why Google can't stop arguing with itself.

TL;DR:

• llms.txt is a Markdown "site tour for AI" at your web root, proposed by fast.ai co-founder Jeremy Howard in September 2024, updated to v2 in August 2026
• Adoption grew 25x in seven months: 10.13% of a broad domain sample, 28% among tech-heavy sites — a good chunk of it generated by default by website platforms
• Ahrefs analyzed server logs from 137,000 domains: 97% of llms.txt files got zero visits all month, and AI never goes looking for files that don't exist
• Google's official guide says Search ignores the file — days later, the Chrome team added an llms.txt audit to Lighthouse anyway
• Bottom line: worth it for developer docs sites; marketing sites can skip it

It's a guided tour, not an instruction

llms.txt is a Markdown file at your web root: one paragraph of introduction plus grouped links that tell AI agents what the site is and where the important pages are — like a map at a park entrance, so an agent doesn't have to walk the whole grounds to find what matters.

Jeremy Howard (co-founder of fast.ai and Answer.AI) proposed it in September 2024; v2 landed in August 2026 with per-page .md versions and link conventions3. The name deliberately echoes robots.txt, but the difference matters: robots.txt is an instruction that compliant crawlers must follow; llms.txt is a self-introduction that no crawler has to read.

Plenty of big names do write one: OpenAI, Anthropic and Google publish llms.txt for their own developer docs, Wix generates one for new sites by default, and WordPress's Yoast plugin ships it as a toggle3. Impressive — until you notice what all those scenes have in common: they are documentation.

25x in seven months, built on speculation

The adoption curve is real. SE Ranking sampled 300,000 domains: in April 2025 only 0.4% of sites shipped llms.txt; seven months later it was 10.13% — a 25x jump1. In Ahrefs' more tech-heavy panel it reaches 28%2. GEO (Generative Engine Optimization) vendors made the file a standard deliverable; site builders made it a default switch.

But what pushes people to write it is speculation, not evidence. Ahrefs' verdict on the adoption wave: it is driven by the bet that "AI platforms may start reading this file," not by any platform confirming that they do2. The platforms haven't announced support; the channels stocked the shelves anyway.

97% of files, zero readers

Ahrefs ran the largest test to date on server logs from its web analytics product: 137,000 domains, observed for the whole of May 20262:

• 28% of domains published an llms.txt; 97% of those files got zero visits — no AI crawler, no human, nothing
• AI never goes looking for llms.txt: on requests to missing files, the AI share was zero; the 404-probers were almost all humans (mostly SEOs checking competitors)
• Among the 3% of files that were fetched, the bulk wasn't AI either: SEO audit tools took 21.7%, and the "llms.txt industry itself" (GEO scorers, validators, directories) another 12% — an industry diligently auditing a file nobody reads
• AI-related requests totaled 19.5%, dominated by training crawlers: GPTBot (model training) led, Claude Code (coding agents) came second — while the AI search bots llms.txt is supposedly for (OAI-SearchBot, PerplexityBot & co.) accounted for 1.1% combined; Slackbot, a chat app's link-preview bot, fetched more than PerplexityBot

Incidentally, Lighthouse — the tool that audits for llms.txt — accounted for roughly one in a thousand of those log entries: 22 requests. Even the checkup tool reads it; just about that enthusiastically.

SE Ranking also ran an attribution check: removing the llms.txt variable from its machine-learning model slightly improved predictions of which sites get cited by AI — as a citation factor, it currently equals zero1.

Google is arguing with itself

The most theatrical moment came within a single week in late May 2026.

First, Google's Search team updated its generative-AI optimization guide, adding to the mythbusting section: you don't need machine-readable files for AI search — Google Search ignores llms.txt entirely; having one won't help, and lacking one won't hurt4. Days later, the Chrome team shipped an "Agentic Browsing" experimental category in Lighthouse with llms.txt among its audits, reasoning that "without it, agents may spend more time crawling your site"5.

When Lily Ray pressed Google's John Mueller on the contradiction, his answer: llms.txt is "not done for search" — it's a "temporary crutch, perhaps to save some tokens" for AI coding tools parsing developer documentation; sites that aren't documentation shouldn't worry about it2.

That explains our PSI result, too: the category counts for no score, and on our site llms.txt was marked "not applicable." The Search team didn't lie; the Chrome team didn't lie. One cares about rankings, the other about agent crawl efficiency — two KPIs, one file.

Should you write one?

The answer splits by site type:

• Developer docs sites (API references, SDK guides): yes. If your readers include coding agents like Claude Code, llms.txt is a table of contents they genuinely follow — the "heaviest use" the spec's own author describes3.
• Marketing sites: skip it. With a 97% zero-visit baseline, expecting AI citations through this file is expecting business from an envelope nobody opens.

What actually decides whether AI cites you is the same boring trio as ever: be crawlable (don't accidentally block AI bots in robots.txt), be legible (semantic HTML and a clean accessibility tree — WebAIM's 2026 report found an average of 56.1 accessibility errors per page, and rising6), and be quotable (concrete numbers, clear positions, paragraphs that can be lifted as single sentences). Get those right and llms.txt is a footnote; get them wrong and an unread letter won't save you.

When does it become worth writing? The day an AI platform officially announces that its crawlers read llms.txt — the file becomes a standard fixture overnight. Until then, it stays an optional line on the checklist.