We measured llms.txt for 90 days: here is what GPT and Perplexity actually fetched

When llms.txt landed as a proposed standard, we did what we usually do before recommending it to clients: we measured it. Over 90 days we served the file on nine production sites ranging from a 12-page B2B site to a 4,000-URL autoblog, instrumented every request, and tracked downstream citations in ChatGPT, Perplexity and Google AI Overviews.

The setup

We exposed /llms.txt with a curated Markdown index per site, kept the existing sitemap and robots.txt untouched, and logged every hit with user agent, timestamp and the requested path. Citation tracking was the messy part: we polled the three surfaces weekly with 40 branded and unbranded queries, then manually verified each attribution by reading the linked source. No vendor dashboards, no proxies.

What actually fetched the file

Of the nine AI crawlers we identified through reverse DNS, only three hit llms.txt more than once a week: GPTBot, OAI-SearchBot and PerplexityBot. ClaudeBot, Bytespider and Google-Extended together accounted for under 4 percent of requests. Two crawlers we expected, CCBot and Cohere's bot, never appeared on any of the nine sites. The average site saw 11 llms.txt fetches per day, with peaks inside 30 minutes of a fresh publish.

Did it move citations?

This is where we have to be careful. Across the nine sites, citations in ChatGPT and Perplexity rose by a median of 18 percent over the 90-day window, but three sites saw no change and one saw a 9 percent drop, which we later traced to a content refresh that happened mid-test. Google AI Overviews barely budged, median lift of 3 percent, which sits inside our noise floor. The 18 percent figure is real but not attributable to llms.txt alone: structured data, internal linking and freshness moved in parallel.

The failure modes we hit

Three things broke. First, a generated llms.txt that included every URL caused two crawlers to throttle us, request rate dropped to near zero for 72 hours. Second, Markdown that mirrored the sitemap verbatim gave crawlers nothing they could not get from sitemap.xml, and citation lift was zero on those sites. Third, we discovered that updating the file too frequently triggered what looked like a soft ban from one crawler, fixed by capping updates at one per six hours.

What we ship now

Our default llms.txt is hand-curated to 20 to 60 entries per site, grouped by intent rather than mirrored from the sitemap. We regenerate it on publish, not on a timer, and we keep it under 200 KB. We still treat sitemap.xml as the source of truth for discovery and llms.txt as a citation hint layer, not a replacement.

The honest takeaway

llms.txt is worth doing on sites that already invest in structured data and freshness, because the lift stacks. On a thin site with no publishing cadence, it does almost nothing, and the engineering time is better spent on schema and internal linking. Measure before you commit, and do not trust any vendor who quotes a citation lift without showing you the query set.