Green while nothing runs: monitoring a pipeline that mostly idles

An availability probe answers one question: does the service respond right now? For a request-serving web service, that is close to the whole job. For a pipeline, it is nearly irrelevant. The Tatano Energy platform publishes 8 SEO articles every day with 0 manual intervention required; between those runs there is nothing to respond to. A green probe during the idle hours proves nothing about whether tonight's work will happen.

Once or twice a minute answers a different question

The SRE guidance is explicit about sampling: for a web service targeting no more than 9 hours aggregate downtime per year (99.9% annual uptime), probing for a 200 status more than once or twice a minute is probably unnecessarily frequent. That interval is derived from an availability target and a unit of failure, and the unit there is a request. In a daily pipeline, the unit of failure is a run or a work item. A per-minute probe oversamples a signal that changes a handful of times a day.

Traffic is a count per window

The four golden signals are latency, traffic, errors and saturation. Traffic survives the move to batch work, but it changes shape: it becomes items finished per window rather than requests per second. The Tatano case gives the expected figures directly — 8 articles a day, 7 languages served, 4 country domains indexed separately. The personal media platform publishes articles daily with a weekly content operations time of ~0 h. Each of those is an expectation you can compare an observed count against.

That comparison is the alert condition. Not a non-200, but a shortfall against the count the schedule implies for the window. It is a less satisfying check to build, because it only means something once a window closes, and it needs the expectation written down somewhere rather than carried in someone's head.

The failure that produces no error

White-box monitoring is what allows detection of imminent problems and failures masked by retries. Absence belongs in the same category, and it is worse instrumented. The crontab manual notes that if the last entry is missing a newline, cron will consider the crontab at least partially broken. A job that never fires writes no error, triggers no retry, and emits no log line. The only observable is a count that did not move.

So the scheduler needs to report on itself. A record written on every tick, before any work is claimed, separates two situations that look identical from the outside: a pipeline that never started, and a pipeline that started and produced nothing. Those have different causes and different fixes, and no downstream metric will distinguish them for you.

Saturation lives in the queue table

Postgres documents SKIP LOCKED as a way to avoid lock contention with multiple consumers accessing a queue-like table, and warns that skipping locked rows provides an inconsistent view of the data. That warning matters once the queue becomes a metric. A row currently being worked on is not a row waiting for a worker. Decide whether pending means unclaimed or unfinished, and store the claim time so a count can tell the two apart.

Latency then splits into two numbers worth keeping separately: how long an item takes from claim to completion, and how long it waits before being claimed. The second is the one that predicts trouble. If waiting time grows while completion time stays flat, the workers are healthy and there are simply not enough of them for the arrival rate. One timestamp at claim and one at finish give you both.

Checks you can derive from the schedule

Three cover most of it. A count of completed items per window, compared against the figure the schedule implies. The age of the oldest unclaimed row, which turns a stalled worker into a number before missing output makes it obvious. And a liveness record written by the scheduler on every tick, so a broken crontab is visible as a broken crontab rather than as a quiet day.

Availability probes are cheap to add, which is why they end up monitoring workloads they were never designed for. Before wiring up another endpoint check, write down the expected count of finished work per window and the acceptable age of the oldest waiting item. Those two numbers describe a system whose normal state is idle. The probe interval then follows from the cadence of the work, not from a habit borrowed from request serving.

Takeaway

If a pipeline's output is a count, its monitoring should be a count. Derive the check interval from how often the work is supposed to happen, keep claim and completion timestamps so waiting is measurable separately from processing, and make the scheduler leave evidence that it ran at all. A dashboard that is green while nothing runs is not reassurance; it is an unanswered question.

Sources

Google SRE Book — Monitoring Distributed Systems — https://sre.google/sre-book/monitoring-distributed-systems/

PostgreSQL 18 Documentation — SELECT (The Locking Clause) — https://www.postgresql.org/docs/current/sql-select.html

man7.org (cronie) — crontab(5) Linux manual page — https://man7.org/linux/man-pages/man5/crontab.5.html

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy

Neurolinks case study — A personal brand that publishes itself — https://neurolinks.be/work/matthieu-pesesse-media