We cut LLM timeouts from 30s to 4s and lost no quality

A B2B chatbot serving four countries started bleeding p95 latency in week six. Cold starts on a mid-tier model were stretching single-turn responses past 28 seconds, and our 30-second timeout was a hair too generous. Users were not waiting, they were reloading, which doubled our upstream cost per session and made our dashboards look like a heart monitor.

The decision we had to avoid

The obvious move was to throw money at it: upgrade the model, add a faster fallback, or pre-warm connections. We had tried all three on previous projects and knew the bill. Instead we asked a sharper question: what is the actual user patience threshold, and what does the latency distribution look like if we cap it aggressively at the edge?

What we measured before changing anything

We logged every request with three fields: time to first token, total duration, and whether the user saw a complete answer or a truncation. Over 72 hours we collected 41,000 sessions. The shape was ugly. 11 percent of responses landed between 20 and 30 seconds, and 4 percent exceeded the timeout entirely. The median sat at 6.4 seconds, which meant most users were fine, but the long tail was poisoning every aggregate metric we reported to the client.

The brutal cut

We dropped the timeout to 4 seconds and added a streaming-first contract: if the model had not produced the first token within 1.2 seconds, we cancelled and returned a graceful retry prompt. Internally we called it the 4-second rule. The team was nervous. Quality regressions on chat products are silent, and silent failures are the ones that show up in churn two quarters later.

The numbers two weeks later

Completion rate moved from 96 percent to 97.1 percent. Counter-intuitive, but true: cutting the timeout forced the model to commit to answers it would otherwise ramble through. Average session length dropped from 4.1 turns to 3.6 turns, which we initially read as a regression until we saw resolution rate climb 8 points. Users were getting answers, not negotiations. Cost per resolved query fell 22 percent.

What we did not measure and wish we had

We did not A/B test a longer timeout against the short one with the same population, so the 8-point resolution lift is confounded with seasonality. We also did not segment by language, and the four-country mix shifted during the test. If we ran it again we would block by locale and run for at least 30 days before calling it.

The pragmatic takeaway

Long timeouts feel safe but they hide model indecision and punish the median user. A short, enforced timeout is a forcing function: it makes you write better prompts, pick faster models for the hot path, and design graceful fallbacks. Measure completion, resolution, and cost per resolved query, not just latency, and you will stop being afraid of the cut.