When pgvector recall falls apart: what we saw at 500k rows
Published · Updated
Our RAG system quietly degraded as the corpus crossed half a million embeddings. Here is the exact threshold, the metric that caught it, and the fix that did not cost us a rewrite.
We had a RAG pipeline that looked healthy for months. Retrieval latency was flat, p95 stayed under 200ms, and the qualitative spot checks from the team kept coming back positive. Then the corpus crossed 500,000 chunks and the answers started getting worse. Nobody noticed for two weeks because the dashboards were still green.
The symptom nobody flagged
The first sign was not technical. A domain expert reviewing a weekly sample of generated answers started marking more responses as 'plausible but wrong'. We pulled the answers, inspected the retrieved chunks, and the chunks looked reasonable. The model was not hallucinating from nothing, it was grounding itself in adjacent but irrelevant material. The retrieval step had quietly lost precision.
The number that explained it
We re-ran a held-out evaluation set of 200 queries with known good chunks. Recall at k=10 dropped from 0.91 on a 200k row index to 0.74 on the 500k row index, using the same HNSW parameters we had tuned six months earlier. The index had grown, the graph had become denser, and our ef_search value was no longer pulling enough candidates. Recall at k=20 held up better, around 0.86, but we were truncating to top 10 downstream.
Why the dashboards lied
Latency stayed flat because HNSW is approximate and the query planner was still doing the same number of graph hops. Recall is a quality metric, not an operational one, and we had never wired it into production observability. Lesson: if your only signal is latency, you will ship a slow degradation of answer quality and call it a win.
What we changed, and what we did not
We did not move to a dedicated vector database. We raised ef_search from 40 to 120, which lifted recall at k=10 back to 0.88 at a cost of about 35ms on p95. We also added a second evaluation pass that samples 50 queries per day from real traffic, retrieves with the production settings, and compares the top chunk against a slower high-recall baseline. Drift above 5% pages the on-call.
The honest trade-off
Doubling ef_search is not free. Index build time went up, and we now pay roughly 30% more in CPU on the retrieval tier. For a 500k row corpus that is fine. For a 5M row corpus we will probably need to revisit the choice between pgvector, a sharded ANN index, or a hybrid that keeps metadata in Postgres and vectors in a purpose-built store. The point is not that pgvector failed, it is that a parameter tuned for one corpus size is not portable to another.
A pragmatic takeaway
If you run RAG on pgvector, treat recall as a first-class production metric, not a one-off benchmark. Re-evaluate whenever your row count roughly doubles, log ef_search and the index size alongside every query, and keep a slow high-recall retrieval path you can compare against in production. The failure mode is silent, and the fix is usually a single knob rather than a migration.
Working on a project where these methods apply?