Prompt caching is not a free lunch: the budget rule we had to add

We shipped prompt caching on a customer-facing copilot and felt clever for about nine days. Cache hit rate sat near 78%, latency dropped, and the dashboards looked healthy. Then the invoice arrived. The bill had roughly tripled against the pre-cache baseline, and nobody had set a guardrail.

What caching actually changes

Prompt caching reduces per-token cost on repeated prefixes, which is genuinely useful for system prompts, retrieved context blocks, and long tool definitions. It does not reduce the number of requests, the size of the context you send, or the tokens the model still has to read and write. If your copilot fans out into multi-step agent traces, you can cache aggressively and still burn budget on the tail.

The failure mode we hit

Our retrieval layer was returning larger and larger context blocks as the index grew. Each request cached cheaply, but the average cached prompt ballooned from about 3.1k tokens to 9.4k tokens over six weeks. Cache hit rate stayed high, so the cost-per-request looked fine in isolation. Multiplied across 40k daily sessions, the cached-read line item quietly overtook the uncached baseline. Nobody got paged because no single metric was red.

The budget rule we now enforce

We treat cached-token spend as a first-class line item with its own weekly cap, currently 1.4x the pre-cache baseline. Above that, an alert fires and a human reviews the retrieval configuration. We also cap the average cached prompt size at 6k tokens and reject retrieval payloads that would push us over. The rule is simple: caching is a multiplier, not a discount, so it needs a ceiling.

Trade-offs we accepted

Tighter context caps mean slightly lower recall on long-tail queries, which we measured at roughly a 3% drop in grounded-answer acceptance. We judged that acceptable against a 60% reduction in worst-case spend. We also disabled caching on tool-heavy agent paths where hit rates were below 20%, because the bookkeeping cost was not worth the marginal saving.

Takeaway

If you ship prompt caching without a budget, you have not saved money, you have added a multiplier to a number you were not watching. Set a cap, instrument cached tokens separately, and review the retrieval payload size on the same cadence as the model itself.