The scratchpad accumulation that broke our agent budget

We built a research agent that iteratively queries APIs, scrapes pages, and synthesises findings. It worked beautifully in staging with three tool calls. In production, tasks averaged 14 calls. The bill quadrupled overnight, and the culprit was our own scratchpad.

The accumulation problem

Each agent step appended the raw tool output to a running scratchpad in the prompt. Early steps returned 200-token JSON payloads. Later steps, hitting verbose endpoints, returned 1,500 tokens each. By step 12, the scratchpad alone consumed 9,200 tokens of context.

Because the LLM must read the entire context on every call, those 9,200 tokens were re-processed at step 13, step 14, and step 15. You pay for the full input window on every iteration, not just the delta. A 14-step run cost 84,000 input tokens before the final synthesis.

Why truncation breaks reasoning

Our first instinct was to truncate the scratchpad, keeping only the last 3,000 tokens. This immediately tanked task completion rates from 78 percent to 34 percent. The agent lost access to critical details gathered in steps 2 and 4, causing it to re-query the same endpoints.

Blind truncation destroys the causal chain. If step 4 extracted a user ID needed for step 12, cutting it forces a cycle. The agent spins, re-requesting the same data, which inflates the scratchpad faster. Truncation did not save money; it increased latency and failure rates.

Rolling summarisation as a budget lever

We replaced raw accumulation with a two-tier memory: a short-term buffer holding the last two raw tool outputs, and a long-term summary. Every four steps, the agent paused to compress the buffer into the running summary using a smaller, cheaper model.

This added an extra LLM call every four steps. However, compressing 4,000 tokens of raw output into 400 tokens of summary saved 3,600 tokens of input cost on every subsequent step. By step 14, the context window held 1,200 tokens of context plus a 1,000-token summary.

The trade-off is measurable. Summarisation costs roughly 1,500 tokens per compression step. For a 14-step run, that is 4,500 overhead tokens. But it prevents the 9,200-token scratchpad from being re-processed 8 times, saving over 70,000 input tokens per run.

Measuring the compression loss

Summarisation is lossy by definition. We evaluated it by checking whether the final output contained the key facts present in the raw tool outputs. We defined a fact as a specific entity or numeric value extracted during the run.

Baseline retention with full scratchpad was 94 percent. Rolling summarisation dropped retention to 88 percent. A 6 point fact retention loss is significant, but it compares favourably to the 44 point completion rate drop we saw with blind truncation. It is a controlled degradation.

We mitigated the loss by tagging critical entities during extraction. Before summarisation, the agent marks high-priority facts. The summarisation prompt is instructed to preserve these tagged entities verbatim. This brought retention back up to 91 percent with zero additional token cost.

The pragmatic takeaway

If your agent loop runs more than six iterations, model the token cost of your scratchpad re-reads. A rolling summarisation tier with entity preservation is a simple, deterministic way to cap input token growth. Measure the fact retention trade-off explicitly, and set your compression frequency based on your specific cost-to-accuracy ratio.

Do not wait for the bill to spike. Profile the scratchpad size at step 5, step 10, and step 15 during your first integration test. If the curve is exponential, implement compression before the agent leaves staging.