Idempotency keys: the boring pattern that stopped our pipelines double-billing clients

Every automation pipeline we ship has retries. Queues redeliver, workers crash mid-task, webhooks arrive twice. That is not a failure mode, it is the normal operating condition. The failure mode is what your code does on the second delivery. Early on, ours did the same thing it did the first time: it ran the side effect again. This article is about the single pattern that fixed it, and what it cost us to learn.

The incident that forced the issue

A scheduled job generated monthly usage invoices for a client's SaaS. One night the worker published the invoice, then died before acknowledging the message. The queue redelivered. A second invoice went out 40 seconds later, same amount, different invoice number. The client's customer paid both. Total cleanup: two refunds, one apologetic call, and about six hours of engineering time to confirm it had not happened elsewhere. The retry mechanism worked exactly as designed. Our handler did not.

The pattern, concretely

Every job now carries an idempotency key: a deterministic string derived from the logical operation, not the delivery attempt. For the invoice job it is client_id plus billing period. Before executing any side effect, the handler checks a processed_keys table. If the key exists, it returns the stored result and acknowledges. If not, it executes and writes the key in the same database transaction as the result. Retry number five behaves identically to attempt number one.

Where the key must live

The common mistake is checking the key in application memory or a cache with eviction. Neither survives a worker restart, which is precisely when duplicates happen. The key store must be as durable as the side effect itself. We use a unique constraint in Postgres because our side effects already write there. One transaction, one commit: either both the effect and the key land, or neither does. That atomicity is the whole point; a check-then-act across two systems just moves the race condition.

What it costs

Honesty: this is not free. Each handler gains a table, a lookup, and a decision about key granularity. Across our pipelines the extra read adds roughly 2 to 5 milliseconds per job, irrelevant next to the hundreds of milliseconds the jobs themselves take. The real cost is design time. You must decide what 'the same operation' means. Is a regenerated PDF the same job? Is a re-sent email with corrected copy? Those are product decisions, and the pattern forces you to make them explicitly.

Keys are not a substitute for exactly-once side effects

Some effects cannot be deduplicated after the fact. A sent email is sent. For those we split the handler: prepare everything, then perform the irreversible step last, guarded by the idempotency check. If the worker dies after the send but before the key write, a duplicate is still possible, so for email we also pass the key to the provider's own deduplication API where one exists. Two layers, because the blast radius of a duplicate client email is larger than the cost of both.

What we measure now

We track duplicate-suppression hits as a first-class metric. In a typical month across our production pipelines, the idempotency layer silently absorbs between 0.3% and 1.1% of job executions, retries that would otherwise have re-run side effects. Before we measured this, we assumed redelivery was rare. It is not rare; it is constant, just usually harmless-looking. The dashboard number is also an early warning: a sudden spike means an upstream worker is crash-looping.

Takeaway

If your automation touches money, email, or anything a client can see, assume every job will run at least twice and design for it. A durable idempotency key, written atomically with the side effect, is a day of work per pipeline and removes an entire class of 2am incidents. Add the metric. The first time you see the suppression counter tick upward and nothing break, you will understand why this pattern is non-negotiable in our stack.