When exact search also misses: triaging a failed retrieval

A retrieval failure arrives as a vague complaint: the assistant answered without the paragraph that contained the answer. The instinct is to go to the index and turn knobs. That is the wrong first move, because at least two different faults produce the same symptom, and one of them is completely immune to index tuning. The cheapest way to separate them is already sitting in the database.

The default is an oracle, not a fallback

By default, pgvector performs exact nearest neighbour search, which provides perfect recall. You add an index to use approximate nearest neighbour search, which trades some recall for speed. Read that sequence in reverse and you have a diagnostic: the unindexed query is not a slow legacy path; it is the ceiling your indexed path is approximating. Keep a way to run it, even if production never does.

Run the failing question twice: once through the production path, once through exact search over the same vectors and the same embedding model. If the missing chunk appears in the exact result and not in the indexed one, the approximation lost it. If it is absent from both, the index is innocent and every hour spent on index parameters will change nothing about this failure.

Two misses that look identical

The first miss is an approximation miss. It belongs to the search configuration: pgvector lets you specify the size of the dynamic candidate list for search, which is 40 by default, and HNSW has better query performance than IVFFlat in terms of the speed-recall trade-off while costing slower build times and more memory. These are levers with a measurable price, and the exact-search comparison tells you whether you need to pull them.

The second miss is a representation miss. The vector simply does not encode what the question asked for. Embedding models excel at capturing semantic relationships but can miss crucial exact matches: a part number, an error code, a surname. No candidate list size recovers a similarity that was never there. Measure both with one metric: 1 minus recall@20, the percentage of relevant documents that fail to be retrieved within the top 20 chunks.

The lexical half of the problem

When the target is a literal string, Postgres has a separate answer. Full-text searching identifies natural-language documents that satisfy a query and optionally sorts them by relevance, using tsvector for preprocessed documents and tsquery for processed queries, with full-text indexing so documents are preprocessed once and searched rapidly afterwards. The pattern operators are not a substitute: the documentation is explicit that LIKE, ILIKE and the regex operators lack many essential properties required by modern information systems.

That gives a second triage question after the first one. If the missing chunk is absent from exact vector search but contains the exact token the user typed, the gap is lexical, and the fix lives in a text search index rather than in the vector index. Two indexes, two failure modes, one evaluation set to tell them apart.

Fix the text before the parameters

If the representation is the problem, the published numbers favour working on what you store. Contextual retrieval can reduce the number of failed retrievals by 49%, and, when combined with reranking, by 67%. Both figures describe changes upstream and downstream of the vector index, not inside it. That is the uncomfortable implication: the largest reported reductions come from editing chunks and reordering results, not from search settings.

Reranking is also a distinct stage with a distinct bill. It runs after retrieval, on a candidate set, so its cost scales with how many chunks you pass it rather than with corpus size. Treat it as a second decision, taken after you know whether contextual chunks alone closed the gap on your evaluation set. Stacked changes with one measurement teach you nothing about which one worked.

Before any of this

Check the size of the corpus first. If your knowledge base is smaller than 200,000 tokens, about 500 pages of material, you can include the entire knowledge base in the prompt, with no need for RAG or similar methods. A retrieval failure in a corpus that small is a self-inflicted failure: there is nothing to retrieve from because nothing had to be left out.

Whichever path you keep, log what came back. The original RAG paper, which paired a pre-trained seq2seq model with a dense vector index of Wikipedia accessed by a neural retriever, set the state of the art on three open-domain QA tasks and produced more specific, diverse and factual language than a parametric-only baseline, while naming providing provenance for decisions and updating world knowledge as open research problems. Untraced retrieval cannot be triaged after the fact.

The practical shape is small. Hold a labelled set of questions with their expected chunks, score it with 1 minus recall@20, and, for every failure, run the exact-search comparison before touching an index. Approximation misses go to the search configuration. Representation misses go to chunking, reranking or a text search index. Writing down which bucket each failure landed in is what stops the next round of tuning from being guesswork.

Sources

pgvector (GitHub) — Open-source vector similarity search for Postgres — https://github.com/pgvector/pgvector

Anthropic — Introducing Contextual Retrieval — https://www.anthropic.com/engineering/contextual-retrieval

PostgreSQL 18 Documentation — Full Text Search: 12.1. Introduction — https://www.postgresql.org/docs/current/textsearch-intro.html

arXiv — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., arXiv:2005.11401) — https://arxiv.org/abs/2005.11401