Retries are traffic, not error handling: budgeting recovery in an unsupervised pipeline

A retry looks like a courtesy extended to a struggling dependency. It is not. Every retry is a fresh request, issued at the worst possible moment, by a client that has already concluded something is unhealthy. The SRE literature is direct about where that ends: a cascading failure is a failure that grows over time as a result of positive feedback. In an automated pipeline, retries are the feedback.

Retries are traffic you authorised

Treat the retry path as a traffic source with its own volume, not as error handling tucked inside a try block. When a dependency slows, attempts stay in flight longer, each one eventually retries, and the retried requests land while the originals are still outstanding. The load that finishes the service off is generated by its own clients, behaving exactly as configured.

This matters more in automation than in interactive systems. A human gives up, closes the tab, goes for coffee. A scheduled worker does not. It retries at the configured rate for as long as the configuration allows, which means an unbounded retry policy is an unbounded load generator pointed at the component least able to absorb it.

A budget makes failing fast the default

The remedy in the SRE guidance is a ceiling the process owns rather than one each call site invents: consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry; just fail the request. The specific figure matters less than the shape, which converts retries into a bounded resource the process spends.

Once the budget is spent, something has to happen other than waiting. A second path is the useful answer. One published platform runs AI-written articles with resilient multi-model fallback across three fallback levels, running daily without supervision. A budget that hands control to a different provider, a cached result or a skipped item degrades the run; a budget with nowhere to go just relocates the stall.

Jitter, and the blips that synchronise everything

Backoff alone is not enough, because fixed backoff preserves the clustering that caused the problem. The guidance is explicit: always use randomised exponential backoff when scheduling retries. If retries aren't randomly distributed over the retry window, a small perturbation such as a network blip can cause retry ripples to schedule at the same time, so the recovery attempts arrive in waves rather than spread across it.

Schedulers manufacture those perturbations too. Under cron, times that occur more than once during the daylight savings time conversion will cause matching jobs to be run twice, and non-existent times during the missing hours will never match, so jobs scheduled then are not run at all. Two copies of a daily job mean two retry loops against the same dependency. Pin the table with CRON_TZ and know which behaviour applies.

Queues under retry pressure

The claim step is where duplicated workers meet. With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped, which avoids lock contention with multiple consumers accessing a queue-like table, at the cost of an inconsistent view of the data. With NOWAIT, the statement reports an error rather than waiting, and errors in a retrying worker become more retries. Under retry pressure, skipping beats erroring.

The signals that surface it

The four golden signals of monitoring are latency, traffic, errors and saturation. Retry storms show up in traffic and saturation before they show up in errors, because the retries are succeeding often enough to mask the underlying fault. White-box monitoring allows detection of imminent problems and of failures masked by retries, which is precisely the state a budgeted pipeline enters just before it exhausts its budget.

External probing will not fill the gap. For a web service targeting no more than 9 hours aggregate downtime per year, which is 99.9% annual uptime, probing for a 200 status more than once or twice a minute is probably unnecessarily frequent. That cadence is fine for availability and blind to retry budget consumption, so instrument the budget itself and alert on the exhaustion, not the symptom.

Three settings carry most of this: a per-process retry ceiling, randomised exponential backoff on every scheduled retry, and a defined action for the moment the ceiling is hit. Add SKIP LOCKED on the queue claim so duplicate workers step around each other, and export retry counts as a metric you can see. None of it prevents the first failure. All of it decides whether the first failure stays the only one.

Sources

Google SRE Book — Addressing Cascading Failures — https://sre.google/sre-book/addressing-cascading-failures/

Google SRE Book — Monitoring Distributed Systems — https://sre.google/sre-book/monitoring-distributed-systems/

PostgreSQL 18 Documentation — SELECT (The Locking Clause) — https://www.postgresql.org/docs/current/sql-select.html

man7.org (cronie) — crontab(5) Linux manual page — https://man7.org/linux/man-pages/man5/crontab.5.html

Neurolinks case study — A personal brand that publishes itself — https://neurolinks.be/work/matthieu-pesesse-media