Cron should enqueue, not execute: putting the work unit in a table

A daily pipeline usually starts as one crontab line that runs the whole job: generate, publish, distribute. That holds until the clock misbehaves or a run overlaps its predecessor. The crontab contract is narrow. It matches a time and starts a process. It carries no record of whether the work is pending, in flight or already finished. Move that record into a table and most scheduling failures stop being outages.

The trigger and the work unit are different things

Cron's documented behaviour is explicit about its edges. Times that occur more than once, during the daylight savings time conversion, will cause matching jobs to be run twice. Non-existent times, the missing hours during the same conversion, will never match, causing jobs scheduled during the missing times not to be run. CRON_TZ specifies the time zone specific for the cron table. None of this is a bug. It is what matching a wall clock means.

If the crontab line executes the pipeline, both edges become production incidents: a duplicate publication or a silent gap. If the line only inserts a row saying work is due, the double fire becomes a duplicate insert the table can reject, and the missing hour becomes a row that never appeared. One is a constraint violation you can ignore. The other is a gap you can see and backfill later.

What the table knows that the scheduler doesn't

A queue-like table lets consumers claim work without colliding. With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped, which is why the documentation offers it to avoid lock contention with multiple consumers accessing a queue-like table. The same documentation warns that skipping locked rows provides an inconsistent view of the data, so this is not suitable for general-purpose work. For claiming, that inconsistency is the feature.

With NOWAIT, the statement reports an error, rather than waiting, if a selected row cannot be locked immediately, which suits a single worker that should fail loudly instead of stepping over contended rows. Either way, the row holds the state, not the process: due, claimed, attempted, done. A second fire finds nothing claimable. An overlapping run finds rows already locked. The scheduler stops being responsible for mutual exclusion.

Retries become rows, and rows can be budgeted

Retries buried in a shell script are invisible. Retries as rows carry an attempt count and a next-attempt time, which makes them schedulable rather than nested. Always use randomised exponential backoff when scheduling retries: if retries aren't randomly distributed over the retry window, a small perturbation such as a network blip can cause retry ripples to schedule at the same time. A column per row is where that jitter lives.

Then cap the total. Consider having a server-wide retry budget, for example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry, just fail the request. Uncapped recovery feeds a cascading failure, which is a failure that grows over time as a result of positive feedback. A queue makes the cap enforceable for the simple reason that every retry is a countable row.

Volume decides whether the migration pays

The cost of moving state into a table is one schema change and a claim query. The benefit scales with how many work units a day the pipeline owes. Tatano Energy runs four markets on one codebase, with 4 country domains indexed separately, 7 languages served, 8 SEO articles published every day and 0 manual intervention required. At that cadence nobody inspects runs by hand, so the record of what is due has to be queryable.

What to watch once the queue is the record

The four golden signals of monitoring are latency, traffic, errors, and saturation. A queue exposes all four with no extra instrumentation: age of the oldest due row, rows created per day, rows in a failed state, rows claimed but never finished. White-box monitoring therefore allows detection of imminent problems, failures masked by retries, and so forth, which is precisely the class of fault that a successful exit code hides from you.

Probing the process also tells you less than reading the table. For a web service targeting no more than 9 hours aggregate downtime per year, probing for a 200 success status more than once or twice a minute is probably unnecessarily frequent. A pipeline that owes work once a day sits further still from that cadence. Check the backlog on a rhythm that matches when rows fall due, not when a prober happens to be cheap.

Start by splitting the line. Let cron insert what is due and let a worker claim it, with SKIP LOCKED if it should move past contention or NOWAIT if it should fail loudly. Give retries randomised backoff and a hard budget. Then alert on the oldest unfinished row instead of the exit status of the last process. The clock will still drift twice a year, and the pipeline will no longer depend on it.

Sources

man7.org (cronie) — crontab(5) Linux manual page — https://man7.org/linux/man-pages/man5/crontab.5.html

PostgreSQL 18 Documentation — SELECT (The Locking Clause) — https://www.postgresql.org/docs/current/sql-select.html

Google SRE Book — Addressing Cascading Failures — https://sre.google/sre-book/addressing-cascading-failures/

Google SRE Book — Monitoring Distributed Systems — https://sre.google/sre-book/monitoring-distributed-systems/

Neurolinks case study — Four markets, one codebase — https://neurolinks.be/work/tatano-energy