Succeeded on the third attempt: making self-recovery visible in a daily pipeline

A pipeline that recovers on its own produces the same artefact whether it worked first time or on the third attempt. The article publishes, the row flips to done, the chart stays green. Google's SRE book names the problem directly: white-box monitoring allows detection of imminent problems and failures masked by retries. Self-recovery is a feature that quietly deletes the evidence of what it recovered from.

Retries hide the thing you need to see

Black-box checks answer one question: did the end state arrive? For an unsupervised publisher, that is the wrong resolution. Eight articles a day can land with every unit succeeding first time, or with most of them clawing through a degraded dependency. Both produce eight finished rows. White-box signals, read from inside the process, are what separate the two cases. Instrument the attempt, not only the outcome.

The queue table already knows

If the work unit lives in a table, the attempt count belongs beside it: how many tries, which error on each, when the last one fired. That turns recovery into data rather than a hole in the log. One caveat on reading it back. With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped, and Postgres documents that this gives an inconsistent view of the data. Workers should claim with it; metric queries should not.

A drained budget is not a green run

The SRE guidance on cascading failures suggests a server-wide retry budget: for example, only allow 60 retries per minute in a process, and if the budget is exceeded, fail the request instead of retrying. Treat that budget as a gauge as well as a cap. A run that finished while consuming most of its allowance is telling you a dependency is degrading, early enough to act before the next run crosses the line.

Backoff changes what the timestamps say

Always use randomised exponential backoff when scheduling retries. The same source explains why: if retries are not randomly distributed over the retry window, a small perturbation such as a network blip can cause retry ripples to schedule at the same time, and a cascading failure is one that grows over time as a result of positive feedback. Randomisation spreads the attempts out, which also means they stop showing up as a visible spike. Count them per unit instead.

Fallback is recovery under another name

The Matthieu Pesesse platform publishes articles autonomously, daily, with three model fallback levels. Fallback masks failure the way a retry does: output appears, and the fact that the first choice failed leaves no trace in the artefact. Record which level served each unit. A week where the third level carried most of the work is a different system from the one that shipped, even though both published daily.

Set the check interval from the cadence

The four golden signals of monitoring are latency, traffic, errors and saturation, and they were written for services answering requests. A job that fires once a day has almost no traffic to watch. The SRE book is also blunt about probe frequency: for a web service targeting 99.9% annual uptime, no more than 9 hours aggregate downtime per year, probing for a 200 status more than once or twice a minute is probably unnecessarily frequent. Scale the same reasoning downward and spend the saved attention on per-unit attempt data.

Pipelines that heal themselves are the goal, not the problem – the Tatano Energy platform publishes 8 SEO articles every day across 4 separately indexed country domains with manual intervention required: 0. Keep that property and still see through it. Store attempts and error reasons next to the work unit, record which fallback level served each one, track retry budget consumption as a trend, and keep locking clauses out of metric queries. Then a green run means what you think it means.

Sources

Google SRE Book — Monitoring Distributed Systems — https://sre.google/sre-book/monitoring-distributed-systems/

Google SRE Book — Addressing Cascading Failures — https://sre.google/sre-book/addressing-cascading-failures/

PostgreSQL 18 Documentation — SELECT (The Locking Clause) — https://www.postgresql.org/docs/current/sql-select.html

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy

Neurolinks case study — A personal brand that publishes itself — https://neurolinks.be/work/matthieu-pesesse-media