Recall@20 is a promise about your prompt: match the metric to what you ship

An eval that reports recall@20 is making a claim about production: that twenty chunks reach the model. The published metric behind contextual retrieval is one minus recall@20, the percentage of relevant documents that fail to be retrieved within the top twenty chunks. If the prompt you ship carries fewer than that, the eval is scoring a retrieval stage that does not exist downstream, and the dashboard answers a question nobody asked.

The metric names a cutoff, not a quality

Recall@20 counts whether relevant documents land inside the top twenty chunks. The useful inversion is one minus that figure, the share of relevant documents that fail to be retrieved, which turns a reassuring high score into a failure rate you can budget against. What it still will not tell you is position inside the window. A document ranked last of twenty and one ranked first both count as a success.

Your baseline is exact search

Before measuring approximation, know what you are approximating. pgvector performs exact nearest neighbour search by default, which provides perfect recall, and an index buys speed by trading some of that away. That gives you a free ground truth: run the eval queries without the index, keep the ordered results, then compare the indexed run against them. And if the knowledge base is smaller than 200,000 tokens, about 500 pages of material, it fits in the prompt and there is nothing to score.

Two settings sit between the cutoff and the answer

Two settings decide whether a reported cutoff describes your system. The dynamic candidate list for search is 40 by default, and it is the pool your top twenty are drawn from. The index type matters too: HNSW has better query performance than IVFFlat on the speed-recall trade-off, at the cost of slower build times and more memory. Record both beside every recall figure, because a number without them is not reproducible.

Failures have a type, not just a rate

A single failure rate hides the part you can act on. Embedding models excel at capturing semantic relationships but can miss crucial exact matches, so a part number or an error code fails for a different reason than a paraphrased question. Postgres ships a separate machine for the first case: tsvector for preprocessed documents, tsquery for processed queries, full-text indexing for rapid search. Split the failures by query type before buying a bigger index.

Reductions transfer only if the cutoff does

Contextual retrieval reduces failed retrievals by 49%, and by 67% when combined with reranking. Both figures are reported against one minus recall@20, so both describe a top-twenty window. Reranking earns its share by reordering candidates inside that window; shrink the window you actually send and you change what the second figure is doing. Reproduce the measurement at your own cutoff before assuming the published gain carries over.

Unsupervised pipelines spend the failure rate quietly

Retrieval quality stops being a lab number once nothing is watching. One personal platform publishes AI-written articles daily with three model fallback levels and roughly zero hours of weekly content operations. Fallback covers a model that will not answer. It does not cover a retrieval that returned the wrong twenty chunks. The original RAG work notes that providing provenance for decisions remains an open problem, which is a reason to log retrieved identifiers per generation.

What to write down

Pick the cutoff from the prompt, not from the paper. Record the number of chunks you send, the candidate list, the index type and the query mix, then report one minus recall at that cutoff as a failure rate. Keep an exact-search run as ground truth while the corpus is still small enough to scan, and log what each generation actually retrieved. The metric then describes your system rather than someone else's.

Sources

pgvector (GitHub) — Open-source vector similarity search for Postgres — https://github.com/pgvector/pgvector

arXiv — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., arXiv:2005.11401) — https://arxiv.org/abs/2005.11401

Anthropic — Introducing Contextual Retrieval — https://www.anthropic.com/engineering/contextual-retrieval

PostgreSQL 18 Documentation — Full Text Search: 12.1. Introduction — https://www.postgresql.org/docs/current/textsearch-intro.html

Neurolinks case study — A personal brand that publishes itself — https://neurolinks.be/work/matthieu-pesesse-media