Queue fan-out vs fan-in: the deadline we almost missed

We had a nightly content pipeline that published across 6 channels for a client. Fourteen jobs, four queues, three workers. It ran fine for weeks. Then a single slow downstream call (an image resize service that started taking 38s instead of 4s) cascaded back through a fan-in barrier and we missed a 6am publication window by 11 minutes. The post-mortem was less about the slow call and more about how we had wired the join.

What fan-out and fan-in actually mean here

Fan-out is the easy part: one trigger splits into N parallel jobs (transcribe, summarise, translate, generate cover image, write captions, schedule). Fan-in is where it gets dangerous. A fan-in barrier waits for all upstream jobs before emitting a single 'publish' event. If any one branch is slow, missing, or silently failing with a retry, the whole pipeline stalls. We had been treating the barrier as a coordination primitive. It is actually a latency primitive, and it inherits the worst-case latency of every branch.

The three failure modes we measured

Over 30 days we logged 47 fan-in stalls. Three patterns dominated. First, retry amplification: a job that fails twice and retries with backoff adds 60-90s before the barrier sees it as done. Second, silent partial completion: one branch succeeded but emitted a malformed payload, so the barrier waited for a timeout instead of failing fast. Third, ordering drift: branches that used to finish in 2-6s started finishing in 15-40s once a downstream API degraded, and the barrier's 60s timeout stopped being generous.

Rule 1: give every branch a hard deadline, not the barrier

We moved timeouts from the fan-in node onto each individual branch. Each job now carries its own deadline (we use 1.5x the p99 from the last 7 days, clamped between 10s and 45s). When a branch exceeds it, the job marks itself as 'failed-fast' and emits a degraded payload (e.g. fallback cover image, machine translation instead of human-reviewed). The barrier then proceeds with what it has instead of waiting for a perfect world. We accept slightly worse outputs in roughly 2% of runs in exchange for never missing the window.

Rule 2: make fan-in idempotent and resumable

Our old barrier assumed each upstream job ran exactly once. In reality, workers crash, messages get redelivered, and a job that 'succeeded' can be re-run after a network blip. We added a content-addressable store keyed on (job_id, branch, content_hash) so the fan-in node can reconstruct the latest state of every branch on restart without re-running anything. This turned a 4-hour manual recovery into a 90-second replay. The cost is one extra Redis lookup per branch, which is negligible.

Rule 3: instrument the slowest branch, not the pipeline

Dashboards that show end-to-end pipeline latency are useless when the pipeline is 14 jobs long. We now expose per-branch p50, p95, p99 and a 'time-since-last-success' gauge per branch. When something degrades, the offending branch lights up immediately. We also alert on 'fan-in wait time' as its own metric, because it is the single number that predicts missed deadlines. If fan-in wait time exceeds 20s for two consecutive runs, we page on it before the window is at risk.

The trade-offs we accepted

Degraded outputs are a real product decision, not just an engineering one. For this client we agreed in advance that a machine-translated caption with a fallback image is acceptable for the 2% of runs where a branch fails-fast, and that we would retry the full pipeline once within 24 hours to replace the degraded version. That contract is what makes rule 1 safe. Without it, failing fast is just failing.

What we would do differently on day one

If we were rebuilding this pipeline today, we would skip the fan-in barrier entirely and use a single orchestrator that polls branch state and emits 'publish' the moment all required branches are present and any optional branches have either succeeded or hit their deadline. The barrier pattern is convenient in tools like Temporal or Step Functions, but it hides the latency composition. Make the latency composition explicit and most of the failure modes become obvious.

The pragmatic takeaway

Fan-in barriers look like coordination but behave like latency. Put deadlines on every branch, make every join idempotent, and instrument branches not pipelines. The goal is not a pipeline that never degrades, it is a pipeline that degrades predictably and never misses the window you actually care about.