Every health check was green while the agent loop had been dead for 9 hours

On a Tuesday morning a client asked why their document-processing agent had produced nothing since 23:40 the night before. Every dashboard was green. The container was up, the HTTP health endpoint returned 200, CPU and memory were normal, and the queue consumer was connected. The system was, by every classical definition, healthy. It had also done zero useful work for nine hours, because the agent loop was stuck waiting on a tool call that would never return.

The failure mode: alive but not working

The root cause was mundane. A retry wrapper around a vector search call caught a timeout exception, logged it, and entered a backoff sleep. A bug in the backoff calculation meant the sleep duration overflowed into hours. The process was alive, the event loop responsive, the health endpoint fine. Classical liveness checks answer the question 'is the process running', which is the wrong question for agent workloads. The right question is 'is the system making progress', and nothing in our stack was asking it.

Why LLM systems fail differently

In a conventional web service, failure correlates with process death or error rates. In agent systems, the most common production failures we see across 14 running systems are stalls: a tool call hanging on a third-party API with no timeout, a model returning malformed JSON that gets retried forever, a queue consumer acking messages but skipping work due to a poisoned branch. In all three cases the process looks perfect to infrastructure monitoring. Error rates stay at zero precisely because nothing completes.

What we measure now: progress, not liveness

We replaced the binary health model with three signals. First, a heartbeat written by the worker itself: every completed unit of work updates a last_progress timestamp in Postgres, and an alert fires if it is older than a per-system threshold, typically 10 to 15 minutes. Second, a throughput floor: if processed items per hour falls below 20 percent of the trailing 7-day median for that hour, we page someone. Third, a stall detector inside the agent loop that hard-kills any tool call exceeding 120 seconds, regardless of what the underlying library thinks is reasonable.

The trade-offs we accepted

Progress-based checks create false positives. A system with genuinely bursty traffic, like the podcast pipeline that runs once daily, will trip a naive throughput alert every quiet hour. We handle this with per-system baselines and quiet-hour windows rather than one global rule, which adds configuration burden: each of the 14 systems carries its own thresholds, reviewed quarterly. We also learned that heartbeats must be written from inside the work loop, not a wrapper around it. Our first attempt put the heartbeat in the consumer's outer loop, which kept beating happily while the inner agent logic was stuck. That version would not have caught the original incident.

What the numbers looked like after

In the six months since the change, the progress checks have caught 11 real stalls across our systems, with a median detection time of 14 minutes. Before, our detection was user-reported, with a median of about 5 hours. False positives run at roughly two per month across all systems, almost all traced to traffic dips we had not modelled. The instrumentation cost was small: the heartbeat is one extra write per work item, and at our volume that is under 50,000 writes a day, negligible against the rest of the workload.

Takeaway

If you run agents or any long-running automation, audit your health checks and ask one question: would this alert fire if the system stayed up but did nothing? If the answer is no, add a progress heartbeat inside the work loop, set a throughput floor against your own historical baseline, and cap every external call with a hard timeout. It is a day of work per system, and it converts your worst class of incident from a client email into a 15-minute page.