The core architectural tension in multi-agent systems is the friction between non-deterministic models and the deterministic orchestration required to run them reliably in production. I routinely see teams wire large language models directly into synchronous application logic. The result is a fragile distributed system where engineers become manual reset switches, untangling runaway loops and recovering lost state.
Autonomous agents cannot scale if they require a human to survive a transient network timeout. To run agents in enterprise environments, you must decouple the model's non-deterministic reasoning from the platform's deterministic state management.
**The synchronous RPC anti-pattern**
Treating an AI agent like a standard remote procedure call (RPC) is an architectural failure condition. When an application calls an agent and blocks a thread waiting for a multi-step run to finish, it assumes the model will return within standard network timeouts. In production, this assumption collapses.
Agents execute loops: they call external APIs, evaluate responses, and self-correct. These are inherently long-running processes. If your orchestration layer holds the agent's context exclusively in memory, a mundane disruption—a pod eviction, a container restart, a network blip—destroys the task's progress. You cannot safely retry from scratch because the agent may have already executed side effects, like mutating a database or hitting a billing endpoint.
Without a durable record of completed steps, developers are forced to manually untangle logs and revert partial state changes. You have turned your engineering team into the disaster recovery mechanism.
**The 3 a.m. failure mode: the unmitigated retry storm**
A primary symptom of lacking durable state is an uncontrolled retry loop that incinerates token budgets and corrupts downstream data.
Consider an agent tasked with migrating customer records: it maps schemas and writes to a target microservice via REST. The agent builds the payload and executes the call. The receiving service experiences a latency spike and takes 15 seconds to respond, but the agent's HTTP client times out at 10 seconds.
The client throws a timeout exception. Architecturally, a timeout is an unknown outcome—the target service may have already committed the write. Because the orchestration layer lacks a deterministic retry policy, it feeds the raw error back into the model with naive instructions to "fix the error." The model, blind to network timing and unaware the write may have succeeded, simply regenerates and resends the identical request. If you haven't implemented idempotency keys, every retry threatens duplicate records. This loop persists until it hits a token budget, a rate limit, or a hard context-length ceiling.
You can detect this in production using standard observability. Watch for rising p99 latency in your orchestration layer correlating with sudden spikes in token consumption and downstream duplicate-key or conflict errors. OpenTelemetry traces will reveal the exact same tool call repeating sequentially without workflow progression. If you see hundreds of failures per minute, you are watching parallel agents or aggressive parallel retries failing concurrently.
**Durable execution and declarative state transitions**
To prevent agents from spinning out of control, implement durable execution—the underlying pattern of workflow engines like Temporal or AWS Step Functions. These engines record the inputs, progress, and results of each step in persistent storage, ensuring the workflow survives process crashes.
Instead of merely shuttling a growing conversation history back and forth, the orchestrator operates as a strict state machine. When the model requests a tool call, the orchestrator logs that intent as an event and executes the tool as an isolated, retryable step. The model calls themselves are recorded steps. Their outputs are saved and replayed on recovery—never regenerated—forcing the orchestration logic to remain deterministic.
If the orchestrator's container dies, the workflow is not lost. A new worker reads the event history, hydrates the state, and resumes from the last checkpoint. Because durable engines guarantee at-least-once execution, a tool call interrupted by a crash will run again. The cost here is strict API contract enforcement: every side-effecting call requires a stable idempotency key (e.g., workflow ID + step ID), and the receiving API must honor it. Durable execution does not eliminate the need for idempotency; it fundamentally depends on it.
A declarative workflow definition makes the allowed transitions explicit:
In this pattern, the orchestrator catches the timeout and handles backoff retries declaratively. The exact same idempotency key accompanies every attempt. If the original write succeeded, the downstream service returns the existing resource rather than creating a duplicate. Only after the deterministic retries are exhausted does the orchestrator prompt the model to reason about the failure or trigger a compensating transaction to rollback previously completed steps. The operational cost of this control is development overhead: you must engineer explicit compensation logic for every mutable action the agent can take.
**The cost of persistent state**
Saving every state transition introduces real operational overhead. You are trading unpredictable application failures for increased database writes and storage consumption.
An event history records each step's inputs and outputs: prompts, raw model responses, tool arguments, and tool results. At enterprise scale, this data footprint grows aggressively. Workflow engines strictly cap payload and history sizes (e.g., 256 KiB per payload in AWS Step Functions, roughly 2 MB in Temporal). The architectural cost is maintaining a claim-check pattern: you must offload large contexts and documents to external object storage and pass only reference URIs through the event history.
Durability also introduces latency. Each checkpoint costs a round trip to the persistent store. While these single-digit or double-digit millisecond hits seem negligible compared to multi-second LLM inference times, they compound rapidly in workflows executing dozens of fast tool calls. If you are on a latency-sensitive, synchronous user path, you must measure this tax rather than assume it is acceptable.
Skip durable execution for simple, single-step tasks. If your model merely extracts structured JSON from a document or classifies an incoming support ticket, synchronous execution is the correct architectural choice. Fail fast and force the caller to retry. Durable state management is strictly for agents executing multi-step workflows with side effects that cannot be safely repeated.
**Moving to asynchronous execution**
Shifting from synchronous request babysitting to asynchronous, durable agent execution forces a paradigm shift in how engineering teams operate. You are no longer writing chat scripts; you are designing asynchronous, event-driven distributed systems. The upfront cost is significant architectural complexity. If your platform cannot currently support durable workflows, strict API idempotency, and compensating transactions, deploying autonomous agents will simply automate your outages at scale.
Comments