The few-shot prompt bloat that broke our semantic cache

We needed stricter output formatting for a client onboarding pipeline. Adding eight few-shot examples to the system prompt pushed classification accuracy from 76% to 90%. Two days later, our semantic cache hit rate dropped from 82% to 11%, and our LLM token bill tripled. The few-shot examples were the culprit.

The symptom: a draining cache

Our semantic cache used embedding similarity to match incoming requests against previous prompts. When the prompt was short, cosine similarity between similar user queries stayed above our 0.92 threshold. Hit rates were high, and LLM calls were cheap. Adding examples changed the vector math entirely.

The mechanism: vector dilution

Few-shot examples add diverse semantic content to the fixed portion of every prompt. When you embed the entire prompt, the specific user query becomes a smaller fraction of the overall vector. The embedding shifts to represent the examples more than the actual intent, pushing similar queries apart in the vector space.

We measured the average pairwise cosine similarity for 500 duplicate user intents. With a bare system prompt, similarity averaged 0.94. With eight few-shot examples appended, it dropped to 0.81. Our 0.92 cache threshold was now completely unattainable for these query pairs, causing the cache miss avalanche.

The fix: query-only embedding

We stopped embedding the full prompt for cache lookups. Instead, we split the template. The system prompt and few-shot examples were stripped before embedding. We embedded only the variable user input for the cache key. This immediately restored our vector representation to the user's actual intent.

Cache similarity for those same 500 intents jumped back to 0.94. Hit rates recovered to 79% within a week. Token costs dropped back to baseline. Classification accuracy remained at 90% because we still sent the few-shot examples to the LLM; we just excluded them from the cache key generation.

The trade-off: context misses

Stripping the system prompt from the cache key introduces a risk. If you change the system prompt, the cache will still serve old answers based on the new user input matching an old key. You must version your cache namespace whenever the system prompt or few-shot examples change, which adds operational overhead.

We handle this by prepending a manual version hash to the cache namespace. If we tweak a few-shot example, we bump the version. It costs us a cold start on the cache for that version, but it prevents stale outputs. It is a manageable trade-off for keeping our token bill stable.

The takeaway

If you use a semantic cache, never embed the full prompt template. Embed only the variable user input, and version your cache namespace to handle template updates. Few-shot examples improve accuracy, but they will destroy your cache hit rate if you let them dictate the embedding vector. Separate your accuracy logic from your cache logic.