The Optional line: deciding what an agent is allowed to skip

A /llms.txt file looks like a five-minute job: one heading, a short list of links, commit. The format really is that simple. The decision underneath it is not. The file is a curated claim about what a model should read to understand your site, and curation does not stay done. The proposal gives you one required section and one convention for everything else, and that convention is where the engineering judgement lives.

One required section, everything else is a choice

The llms.txt proposal is explicit about its own minimum: an H1 with the name of the project or site, described as the only required section. Nothing validates the rest. No tool will tell you that a link is missing, stale, or pointing at a thin page. Compare that with structured data, where you must include all the required properties for an object to be eligible for appearance in Google Search with enhanced display. One format has a pass-or-fail gate. The other has only your taste.

The Optional line is a context budget

The proposal reserves an Optional section, by convention, for secondary information: links an agent can skip when a shorter context is needed. Read that literally and the file stops being a sitemap and becomes a priority order under truncation. The useful test is blunt: if a reader sees only what sits above the line, does it answer what you do, for whom, and with what evidence? Archives, announcements and tangents belong below it.

Cadence breaks the list, not the format

Publishing volume is what actually defeats a hand-maintained list. Our Tatano Energy case study records four country domains indexed separately, seven languages served, eight SEO articles published every day, and zero manual intervention required. No human is going to keep a curated link list in step with that, and a file nobody updates is worse than no file: it describes a version of the site that no longer exists.

Two ways out. Generate the file from the same source the pages come from, so it inherits the cadence instead of fighting it. Or keep the curated part at the level of hubs and category pages, which change slowly, and let daily output be reached through those. Either way, decide per domain and per language rather than once, because four domains and seven languages do not collapse into a single list.

The same-URL markdown is the real commitment

The second half of the proposal is the expensive half, and it gets quoted less. It suggests that pages with information agents might need provide a clean markdown version of those pages at the same URL as the original page. That is not one file; it is a rendering duty on every page you nominate, and the markdown has to keep agreeing with what renders. Put that cost in the platform, where templates live, not in content operations.

What the file does not buy you

Google's own guidance is direct: you don't need to create new machine-readable files, AI text files, or markup to appear in these features. Eligibility to be shown as a supporting link in AI Overviews or AI Mode is indexing, snippet eligibility and the Search technical requirements. The documented control surface elsewhere is robots.txt: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and for search results it can take ~24 hours from a robots.txt update for OpenAI's systems to adjust.

Spend where a number has been measured

There is a measured figure in this area, and it is not about text files. Rotten Tomatoes added structured data to 100,000 unique pages and measured a 25% higher click-through rate for pages enhanced with structured data, compared to pages without it. The same guidance sets limits worth copying: don't create blank or empty pages just to hold structured data, and don't add structured data about information that is not visible to the user.

Treat llms.txt as a cheap statement of priority, not a ranking lever. Write the H1, put a small number of durable pages above the Optional line, and only promise same-URL markdown for pages whose templates can actually deliver it. Then spend the remaining effort where the documentation points: indexable pages, snippet eligibility, and fewer but complete and accurate properties rather than every possible one filled in badly.

Sources used

Sources

llmstxt.org — The /llms.txt file, v2 — https://llmstxt.org/

Google Search Central — AI features and your website — https://developers.google.com/search/docs/appearance/ai-features

Google Search Central — Introduction to structured data markup in Google Search — https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data

OpenAI — Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy