GEO · AI Search · SEO

~24 hours and a navigational link: what a robots.txt edit actually changes

Published · Updated

OpenAI's crawler documentation gives a robots.txt edit a propagation window and a partial effect: ~24 hours for search results, and opted-out sites can still appear as navigational links.

A robots.txt edit feels like a switch. The documentation describes something slower and less total. OpenAI names two separate tags, states a propagation window for one kind of effect, and says that an opted-out site can still appear in one form. If the plan is to add a line and then check whether anything changed, the part that needs planning is the gap between the commit and anything observable.

The edit starts a window, not an effect

OpenAI's crawler documentation states that for search results it can take ~24 hours from a site's robots.txt update for their systems to adjust. Treat the deploy as the beginning of an interval rather than the moment of change. Observations made inside that interval carry no information either way. Record the deploy timestamp somewhere durable, because the later argument about whether the edit worked will turn on when it landed.

Opted out is not absent

The documentation is specific about what opting out of OAI-SearchBot does: sites that are opted out will not be shown in ChatGPT search answers, though they can still appear as navigational links. That is a change in the kind of appearance, not its disappearance. Anyone expecting the domain to vanish is contradicted by the vendor's own page before any crawler is involved, so settle that expectation before the pull request.

Two tags, two different expectations

OAI-SearchBot and GPTBot are described as separate robots.txt tags for managing how a site and its content work with AI. Disallowing GPTBot indicates that a site's content should not be used in training generative AI foundation models. The ~24 hours figure is stated for search results. The documentation does not offer an equivalent observable turnaround for the training decision, so do not borrow one timeline to justify the other.

Coming back is not the same edit reversed

Removing a disallow line restores permission, not presence. On Google's side, the stated precondition is independent of any AI-specific file: to be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. Google also states you do not need to create new machine-readable files or markup to appear in these features.

Nothing reports the thing you changed

Google's AI features are reported in the Performance report within the "Web" search type. That is Google's surface. No dashboard reports whether ChatGPT search answers cite a domain, so verification of an OAI-SearchBot decision reduces to what you control: the file served at the live domain, and the documented interval. Where one codebase serves separately indexed country domains, as on the Tatano Energy platform, each file served starts its own window.

Write the expected difference before the edit

Before changing a line, write the sentence describing what should be different afterwards, and which kind of outcome it belongs to: citation inside answers, navigational links, or use in training. If the sentence cannot be written, the edit is not ready for review. The sentence is also what a colleague will read later when someone asks why the line is there.

An opt-out is cheap to type and slow to confirm. Keep the deploy timestamp, the file as served per domain, and one written sentence of expected effect together in the same place. Wait out the documented interval before concluding anything, and expect navigational links to persist after an OAI-SearchBot opt-out. Where a page needs to be eligible again, the work is the Search technical requirements, not a new file.

Sources

OpenAI — Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots

Google Search Central — AI features and your website — https://developers.google.com/search/docs/appearance/ai-features

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy

Working on a project where these methods apply?