AI agents in production: what actually breaks (and how to prevent it)

An AI agent that has run every night for six months no longer resembles its demo. What keeps it alive isn't the model — it's everything around it: the queues, the fallbacks, the logs, the thresholds.

Failure #1: the provider degrades

Model APIs go down, slow down or change behavior without notice. The answer: multi-level model fallback. When the primary model degrades, the system switches to the next one without interrupting the task — our production harnesses ship with three fallback tiers.

Failure #2: the output overflows

Truncated responses, invalid JSON, context overruns: every model output must be schema-validated before use, and every task must be replayable. A persistent queue (Redis) with error recovery turns an outage into a mere delay.

Failure #3: the agent does what you said, not what you meant

The most expensive risk isn't technical: it's an agent that publishes, sends or deletes too much. Committing actions — public publication, spending, deletion — go through approval modes or confidence thresholds. Autonomy is earned in measured steps, never granted by default.

The three-logs rule

Every production agent keeps three logs: what it decided, what it executed, what it cost. Without those traces, the first incident becomes an investigation; with them, it's a five-minute read.