GPTBot or OAI-SearchBot: two robots.txt lines, two different decisions

Most GEO work is framed as production: sharper answers, cleaner markup, more pages. Access comes first. For ChatGPT, two robots.txt tags decide what a crawler may do with your site, and they answer two separate questions rather than one. Confusing them costs citations quietly, because nothing breaks and no error appears in any report. Here is what the two lines actually control.

Two names, two questions

OpenAI documents two crawler names. GPTBot and OAI-SearchBot exist so that webmasters can manage how their sites and content work with AI. Disallowing GPTBot indicates that a site's content should not be used in training generative AI foundation models. Opting out of OAI-SearchBot has a different effect: sites that are opted out will not be shown in ChatGPT search answers. One decision concerns future models, the other concerns today's answers.

Opting out of search is not disappearance

The search opt-out is not total. Sites opted out of OAI-SearchBot can still appear as navigational links, so the brand does not vanish from the product. That asymmetry matters when someone argues that a blanket disallow had no visible cost: navigational appearances can continue while answer citations stop. Decide which of the two outcomes you are trying to keep, then write the rule for that one rather than for both at once.

Treat the file edit as a deploy

A robots.txt edit is not an instant switch. For search results, it can take roughly 24 hours from a site's robots.txt update for OpenAI's systems to adjust. An afternoon of toggling proves nothing, and a revert made an hour after a block does not restore the previous state straight away. Record the date and time of every change to that file, because the timestamp is the only anchor available afterwards.

Google's side is eligibility, not a toggle

Google states that you do not need to create new machine-readable files, AI text files, or markup to appear in these features. Eligibility to be shown as a supporting link in AI Overviews or AI Mode requires that a page be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. The work there is ordinary indexability and snippet eligibility, not a special file.

Indexability has a serving precondition, and it is cheap to verify. Abbys Consult launched with time to first byte under 200 ms and 0 technical SEO issues. Those are not AI-specific numbers; they are what being indexed and snippet-eligible rests on. Before negotiating crawler access, confirm the page is reachable, fast and clean, because an access policy applied to a page that cannot be indexed changes nothing at all.

The number lands under Web

Measurement is coarser than the decisions you are making. Google reports these features in the Performance report, within the Web search type. There is no separate line to isolate, so a crawler policy change on one platform and a markup change on another both land in the same aggregated data. That is why dated change notes matter more here than dashboards: the report will not tell you which edit moved what.

Four domains mean four robots.txt files

Access policy is per host, so it multiplies with your footprint. Tatano Energy runs four country domains indexed separately, serving 7 languages from one codebase, publishing 8 SEO articles every day with 0 manual intervention. One codebase does not imply one robots.txt: each domain serves its own. Any crawler decision has to be replicated four times, and drift between those files stays invisible until citations start differing by market.

Write the two rules separately and date the change. Decide whether you want to be cited in ChatGPT search answers, and decide whether you want your content used in training; those are different lines with different consequences. Allow around 24 hours before reading anything into the result. On Google's side there is no file to add, only indexing, snippet eligibility and the technical requirements behind them. Then check every domain you own, not just the main one.

Sources

OpenAI — Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots

Google Search Central — AI features and your website — https://developers.google.com/search/docs/appearance/ai-features

Neurolinks case study — A consulting firm, dressed for trust — https://neurolinks.be/work/abbys-consult

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy