The constrained sampling trap that halved our completions

When we deployed a grounded product description generator for a retail client, brand consistency was the top priority. We set temperature to zero and top_p to 1.0, expecting deterministic, on-brand outputs. Instead, we watched our usable completion rate drop from 82% to 41% overnight.

The zero-temperature mirage

At temperature zero, the model always selects the highest probability token. This feels safe for brand control because outputs become highly predictable. The problem is that predictable does not mean correct. It means the model gets stuck in local optima.

We saw the same opening sentence structure repeated across hundreds of generations. Worse, when the highest probability path led to a factual dead end, the model could not escape it. It would confidently hallucinate product specifications rather than deviating to a safer phrasing.

How repetition penalties made it worse

Our first fix was adding a repetition penalty of 1.2. This stopped the verbatim loops but introduced a different failure mode. The model began selecting increasingly obscure tokens to avoid the penalty, producing awkward phrasing that violated the brand voice guidelines entirely.

Usable output recovered slightly to 53%, but review time tripled. Evaluators had to hunt for subtle semantic errors instead of obvious loops. The repetition penalty traded a visible failure for an invisible one, which is far more dangerous in a production pipeline.

The low-temperature sweet spot

We ran a controlled sweep across 4,800 generations at temperature settings from 0.0 to 0.7. The optimal brand consistency score arrived at 0.15, with a top_p of 0.85. This narrow window allowed just enough stochastic escape velocity to avoid local optima.

At temperature 0.15, our usable completion rate returned to 78%. Brand consistency measured by our rubric actually improved compared to zero, climbing from 64% to 81%. The slight randomness prevented the model from fixating on bad paths while keeping the distribution tight enough for control.

The sampling budget trade-off

There is a computational cost to this approach. Low but non-zero temperatures require full softmax calculations across the vocabulary. At zero, frameworks like vLLM can optimise by selecting the argmax directly, skipping the normalisation. Our inference latency increased by roughly 12%.

For most generative pipelines, that 12% latency tax is well worth paying. The alternative is sending unreviewed, confidently wrong content to production, or paying humans to review and rewrite half your outputs. We accepted the latency and optimised elsewhere in the stack.

What to measure before you freeze

Before setting temperature to zero, measure your usable output rate, not just brand consistency. A model that repeats the same safe phrase is perfectly consistent but completely useless. Track semantic diversity alongside constraint adherence to find the actual operating point.

Start at zero, but sweep upwards in 0.05 increments. Plot your brand score against your usability score. The intersection where both metrics hold up is rarely at absolute zero. It is usually just enough randomness to keep the model from trapping itself in its own confidence.