Saturation is the only golden signal a daily pipeline can't borrow

The four golden signals of monitoring are latency, traffic, errors and saturation. They were written for a serving system answering requests it did not schedule. A pipeline that publishes 8 SEO articles a day across 4 country domains schedules its own work, so three of those signals arrive distorted, and one of them does almost all the useful work. Saturation is the signal worth instrumenting first, and the one most batch pipelines never record.

Latency, traffic and errors arrive distorted

Latency in a serving system is time to respond. In a queue-backed pipeline the equivalent is the age of a work unit between enqueue and completion, which can grow for hours without a single call timing out. Traffic is not demand; it is your own schedule: 8 a day is a number you chose, so a traffic chart mostly plots the crontab. Errors still count, but they count late.

Errors count late because retries absorb them. The SRE book is explicit that white-box monitoring allows detection of imminent problems and failures masked by retries. A run that succeeds after retrying is a success at the edge and a problem inside. If the only error metric is the final outcome of the day's batch, the pipeline reports health right up to the point where it can no longer catch up.

Saturation is where the feedback loop starts

A cascading failure is a failure that grows over time as a result of positive feedback. In a scheduled pipeline the loop is plain: work arrives on a fixed cadence, clears more slowly than it arrives, and each day's backlog competes with the next day's enqueue. Nothing is down. Per-call latency is unchanged. The queue is simply longer than it was yesterday, and the gap compounds without an obvious incident to attach it to.

That makes saturation the only signal with lead time. Measure two things on the work table: how many units are waiting unclaimed, and the age of the oldest one. A count alone hides a stuck row behind healthy throughput; an age alone hides a growing pile of fresh work. Together they separate a pipeline that is busy from a pipeline that is behind, which is the distinction a daily cadence depends on.

The queue table measures itself, with one caveat

Workers claiming rows with SKIP LOCKED get what the documentation promises: any selected rows that cannot be immediately locked are skipped. The same documentation warns that skipping locked rows provides an inconsistent view of the data, so it is not suitable for general-purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table. A worker's claim query is a dispatcher, not a census. Do not read backlog figures from it.

Record saturation from a read that does not compete for locks, and keep it off the claim path entirely. NOWAIT is the other contract on offer: the statement reports an error, rather than waiting, if a selected row cannot be locked immediately. That is useful for detecting contention at the moment it happens, and it is still not a measurement of depth. Keep the two concerns in separate queries with separate purposes.

Retries hide saturation, then feed it

Retries are the mechanism that turns a saturated queue into a cascading one. Always use randomised exponential backoff when scheduling retries. If retries are not randomly distributed over the retry window, a small perturbation such as a network blip can cause retry ripples to schedule at the same time. A platform spanning 4 country domains and 7 languages has enough parallel work for those ripples to arrive as one self-inflicted spike.

Cap recovery so it cannot outgrow the work. The SRE book suggests a server-wide retry budget: for example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, do not retry, just fail the request. A budget also converts retries into a signal. Rising budget consumption while completed publications stay flat is saturation reported early, before the backlog itself is large enough to see.

Zero hours of operations is a monitoring specification

The Tatano Energy platform reports manual intervention required: 0, across 4 country domains, 7 languages and 8 SEO articles published every day. The personal publishing engine reports weekly content operations time of about 0 h, with 3 model fallback levels and articles published autonomously, daily. Both figures mean the same operational thing: nobody is watching the queue. What is not recorded is not noticed, and a slow backlog is the least visible failure mode on the list.

External availability checks will not cover it. The guidance for a web service targeting no more than 9 hours aggregate downtime per year (99.9% annual uptime) is that probing for a 200 success status more than once or twice a minute is probably unnecessarily frequent. That cadence is reasonable for a page and blind to a queue. The page can return 200 for weeks while the day's units sit unclaimed behind a worker that quietly stopped claiming.

Instrument the work table before adding another dashboard: unclaimed count, age of the oldest unclaimed row, retry budget consumption and completed units per scheduled cycle. Alert on the age, since it is the figure that moves before anything breaks. And if the response to that alert is mechanical, write it as code instead: the SRE book's own test is that if a page merely merits a robotic response, it shouldn't be a page.

Sources

Google SRE Book — Monitoring Distributed Systems — https://sre.google/sre-book/monitoring-distributed-systems/

Google SRE Book — Addressing Cascading Failures — https://sre.google/sre-book/addressing-cascading-failures/

PostgreSQL 18 Documentation — SELECT (The Locking Clause) — https://www.postgresql.org/docs/current/sql-select.html

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy

Neurolinks case study — A personal brand that publishes itself — https://neurolinks.be/work/matthieu-pesesse-media