200,000 tokens or an index: measure the corpus before you build retrieval

Teams building semantic search usually open with a question about indexes: HNSW or IVFFlat, which distance function, what candidate list size. There is a cheaper question that comes first, and it sometimes removes the index entirely. How large is the corpus, counted in tokens? Anthropic's guidance puts a usable line at 200,000 tokens, roughly 500 pages. Below it, retrieval may be solving a problem you do not have.

The line is drawn in tokens, not documents

The guidance is specific: if your knowledge base is smaller than 200,000 tokens, about 500 pages of material, you can include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods. That reframes the first task. Count tokens across the content you would genuinely index, not files in a folder, and not the pages nobody would ever want cited back to them.

Multilingual estates make that count less obvious than it looks. Tatano Energy serves seven languages across four separately indexed country domains from a single codebase, so the same product knowledge exists in several parallel forms. Whether those forms count as one corpus or seven changes the token total, and the total is what decides whether a retrieval layer is load-bearing or decorative.

pgvector's default already gives perfect recall

If you do build retrieval, notice what pgvector does with no configuration at all. By default it performs exact nearest neighbour search, which provides perfect recall. You add an index to use approximate nearest neighbour search, which trades some recall for speed. The index is therefore a latency and cost decision taken on top of an already correct baseline, not the component that makes search work in the first place.

That ordering matters when choosing between index types. An HNSW index creates a multilayer graph and has better query performance than IVFFlat in terms of the speed-recall trade-off, but it has slower build times and uses more memory. For a corpus rebuilt frequently, build time is an operational cost paid repeatedly rather than once during setup, and memory is a number someone has to provision.

Corpora that cross the line while you watch

A static estate sits on one side of the threshold for years. A corpus fed by a publishing pipeline does not. Tatano Energy publishes eight SEO articles every day, driven by search trends, with zero manual intervention required. Anything accumulating at that rate moves steadily towards and then past 200,000 tokens, which means the architecture decision has an expiry date rather than an answer.

So treat the token count as a monitored number instead of a one-time measurement. Pick the trigger in advance: a scheduled recount, or a threshold alert on the table holding the content. The useful consequence is that you can ship whole-corpus prompting deliberately, knowing which event forces the move to an index, rather than discovering the limit through silently truncated context in production.

What retrieval buys once you need it

Crossing the line is not the only reason to retrieve. The original RAG work pairs a pre-trained seq2seq model as parametric memory with a dense vector index of Wikipedia as non-parametric memory, accessed with a pre-trained neural retriever. It set the state of the art on three open-domain QA tasks and generated more specific, diverse and factual language than a parametric-only baseline.

The same paper is candid about what it did not settle: providing provenance for decisions and updating world knowledge remain open research problems. An index you can write to every day addresses the second more directly than a model you cannot retrain. Provenance stays yours to engineer, which is a concrete argument for recording the identifier of every chunk that reaches the prompt, not just its text.

One more constraint belongs in the same decision. While embedding models excel at capturing semantic relationships, they can miss crucial exact matches. Postgres ships the other half already: full-text search identifies natural-language documents satisfying a query and optionally sorts them by relevance, with tsvector for preprocessed documents and tsquery for processed queries. The regex and LIKE operators lack many essential properties required by modern information systems.

Measure improvements against a baseline you chose

Retrieval improvements are quoted against retrieval baselines. Contextual retrieval reduces the number of failed retrievals by 49%, and by 67% when combined with reranking, evaluated as 1 minus recall@20: the percentage of relevant documents that fail to be retrieved within the top 20 chunks. That metric exists only because something can fail to be retrieved. Put a small corpus entirely in the prompt and the failure mode disappears along with the gain.

The sequencing is mundane, which is the point. Count the tokens in the corpus you would actually index. If it sits well under 200,000, put it in the prompt and spend the saved effort on provenance and evaluation. If it is above, or a daily pipeline will take it there, start with exact search for a correct baseline and add an index when latency, not accuracy, becomes the complaint.

Sources

pgvector (GitHub) — Open-source vector similarity search for Postgres — https://github.com/pgvector/pgvector

arXiv — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., arXiv:2005.11401) — https://arxiv.org/abs/2005.11401

Anthropic — Introducing Contextual Retrieval — https://www.anthropic.com/engineering/contextual-retrieval

PostgreSQL 18 Documentation — Full Text Search: 12.1. Introduction — https://www.postgresql.org/docs/current/textsearch-intro.html

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy