One repo, four robots.txt: which GEO controls multiply with domains

Tatano Energy runs as one codebase deployed across four country domains — .lu, .be, .fr, .ch — serving 7 languages and publishing 8 SEO articles every day with 0 manual intervention. That shape is efficient for features and misleading for AI search. Templates are written once. Several of the controls that decide whether an assistant can cite you are not in the templates at all: they sit at the root of each host.

Four hosts mean four robots.txt files

OpenAI exposes two robots.txt tags, OAI-SearchBot and GPTBot, to let webmasters manage how their sites and content work with AI. robots.txt is a per-host resource, so four country domains mean four copies of that decision, and nothing stops them drifting. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Disallowing GPTBot indicates content should not be used to train generative foundation models.

Those are different decisions with different consequences, and a deploy step that copies one robots.txt to every target makes them once for every market at once. Write the file per host deliberately, or generate it from per-domain configuration and keep the generated result readable. Then budget for the lag: for search results, it can take ~24 hours from a site's robots.txt update for OpenAI's systems to adjust.

The llms.txt proposal has the same shape. It proposes a Markdown file at /llms.txt to provide LLM-friendly content, and the only required section is an H1 with the name of the project or site. One host, one file. Four hosts, four files, each naming a different domain and, where markets diverge, different content. The proposal also describes an Optional section by convention, for links an agent can skip when a shorter context is needed.

Page-level signals live in the template

Structured data behaves differently. All required properties must be present for an object to be eligible for enhanced display in Google Search, and the guidance is explicit that fewer but complete and accurate recommended properties beat every possible property supplied badly formed or incomplete. Those rules apply per page, which means in practice they apply per template. Get the article template right once and 8 publications a day inherit it across every domain the codebase serves.

The constraint that bites in an automated pipeline is visibility: don't add structured data about information that is not visible to the user, and don't create blank or empty pages just to hold it. A generator that fills a property from a field the layout never renders breaks that rule on every page it emits, across every domain, and it breaks silently because the markup itself is well formed.

Eligibility for AI features is page-level too. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. In the Tatano deployment, the country domains are indexed separately, 4 of them. Anything that suppresses snippets or indexing on one host removes that market from the feature and leaves the other three untouched.

What multiplies is cost, not capability

Google states that you don't need to create new machine-readable files, AI text files, or markup to appear in these features. That reframes the host-level column: llms.txt is a proposal, and maintaining one per domain is a cost you choose to take on. Indexing and snippet eligibility are not optional, and they already travel with the templates and headers you ship rather than with the number of deploy targets.

The published evidence for markup is a volume argument. Rotten Tomatoes added structured data to 100,000 unique pages and measured a 25% higher click-through rate for enhanced pages compared with pages without it. That is a corpus effect, not a per-page trick. A pipeline publishing 8 articles a day in 7 languages accumulates a corpus the same way, which is the argument for putting markup in the generator rather than in a manual pass.

Fan-out is answered per market

Both AI Overviews and AI Mode may use a query fan-out technique, issuing multiple related searches across subtopics and data sources to develop a response. Subtopics are a content question, not a markup one. With four separately indexed domains and 7 languages, the subtopic surface differs per market, and coverage you hold in one language does nothing for a domain that never published the equivalent page.

Measurement stays blunt. These features are reported on in the Performance report, within the Web search type, so there is no separate AI Overviews line to read. Four separately indexed country domains at least give you four sets of host-level numbers, which makes a change in one market legible against the other three instead of being averaged into a single aggregate you cannot decompose later.

A short inventory before the next deploy

Sort every control by the surface it applies to. Per host: the OAI-SearchBot and GPTBot robots.txt tags, and llms.txt if you publish one. Per template, and therefore per page: required structured data properties, the visibility rule, snippet and indexing eligibility. Per market: the subtopics you actually cover in that language. The first list grows with deploy targets. The second and third do not.

Count the hosts before planning GEO work. A shared codebase makes page-level signals cheap to fix and host-level files easy to forget, and the forgettable column carries the opt-out decisions with the longest tail. A wrong robots.txt line on one domain does not show up in a template diff, takes ~24 hours to take effect for search results, and can remove one market from ChatGPT search answers while the other three keep working normally.

Sources

llmstxt.org — The /llms.txt file, v2 — https://llmstxt.org/

Google Search Central — AI features and your website — https://developers.google.com/search/docs/appearance/ai-features

Google Search Central — Introduction to structured data markup in Google Search — https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data

OpenAI — Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy