Agent Reliability Is Mostly Plumbing

The glamorous part of an agent system is the reasoning loop.

The production part is everything around it.

A model decides what to do next. That matters. But the difference between a demo and a system people can trust usually comes from less dramatic machinery: queues, leases, retries, idempotency keys, timeouts, structured outputs, snapshots, validation gates, approval boundaries, and backpressure.

That is the uncomfortable lesson of reliable agents. The model can be capable and the product can still be brittle if the plumbing is weak.

The Agent Is Not the Whole System

It is tempting to treat the agent as the unit of reliability.

Did it choose the right tool? Did it write the right answer? Did it recover from the error? Did it follow the instruction?

Those questions are important, but they are incomplete. An agent run is embedded in a larger operating system. Work arrives somewhere. State is loaded from somewhere. Tools mutate something. Side effects need boundaries. Partial progress needs to be preserved. Failures need to be retried or recovered without duplicating work. Humans need a place to approve, reject, or resume.

If that surrounding system is loose, the agent has to compensate for too much. It has to remember which operation already ran. It has to infer whether a timeout means the tool failed or only the response was lost. It has to decide whether it is safe to run the same write again. It has to notice that another worker is touching the same task.

That is too much trust to place in a prompt.

Good plumbing narrows the problem. It gives the agent a stable surface to operate against, and it gives the operator evidence when something goes wrong.

Queues Make Work Explicit

A queue is one of the simplest reliability tools an agent system can have.

Without a queue, work often lives inside a request, a chat message, a cron tick, or an in-memory process. If the process dies, the work disappears. If two workers wake up, they may duplicate the work. If a dependency slows down, the system either blocks the user or starts improvising around a hidden backlog.

A queue turns work into an object with state.

The task is pending. The task is leased. The task is running. The task is waiting for approval. The task failed with recoverable evidence. The task is complete. The task is dead-lettered because continuing would be unsafe.

That state gives the agent a contract. It also gives the surrounding system a way to apply policy: concurrency limits, priority, deadlines, retry caps, escalation paths, and human review.

For production agents, this is not just infrastructure hygiene. It changes behavior. The agent no longer has to pretend every request is a fresh conversation. It can operate inside a durable work item that records what has happened and what is still allowed.

Timeouts Are Product Decisions

A timeout looks like a low-level setting until it fires.

Then it becomes a product decision.

How long should an agent wait for a browser action? How long should it hold a lock on a repository? How long should a user wait before the run becomes background work? When should a slow dependency trigger a retry? When should it trigger recovery? When should the system stop because the remaining action might create a duplicate side effect?

If those answers are not designed, the defaults usually win. A tool hangs. A run keeps waiting. A worker dies while holding ambiguous state. A user retries manually. Two attempts cross in the middle.

Reliable agents need timeouts at multiple levels:

• Tool-level timeouts so a single operation cannot freeze the run.
• Step-level timeouts so the agent can recover from a stuck phase.
• Lease timeouts so abandoned work becomes visible again.
• User-facing timeouts so long work moves into an honest background state.
• Approval timeouts so blocked tasks do not look like active progress forever.

The exact values matter less than the fact that the system has made the choice explicitly.

A timeout should not merely kill work. It should produce evidence the recovery path can use.

Locks Prevent Polite Corruption

Agents are good at acting confident while touching shared state.

That makes locks boring and essential.

If two runs edit the same file, process the same account, publish the same draft, or repair the same incident, the failure may not look like a crash. It may look like polite corruption: each run did something reasonable, but the combined result is wrong.

A lock does not have to be fancy. It can be a row-level lease, a filesystem guard, a repository branch policy, or an application-level mutex around a specific resource. The important part is that the system knows which resources require exclusive access and what happens if the lock cannot be acquired.

The agent should not have to infer that from vibes.

If a post is already being edited, the run should stop or branch. If a customer workflow is already in progress, the next run should attach to it or wait. If a deployment is active, a second deployment should be rejected or queued. If a tool cannot guarantee exclusive access, the agent should treat the operation as higher risk.

Locks are not about slowing the system down. They are about preventing two individually plausible runs from creating one unreliable outcome.

Idempotency Is How Retries Stay Safe

Retries are necessary. They are also dangerous when the operation has side effects.

If an agent sends the same email twice, creates two tickets, charges twice, posts twice, or applies the same migration twice, the retry succeeded in the worst possible way. The system recovered from a transient failure by creating a user-visible failure.

Idempotency is the guardrail.

Every side-effecting operation should have a way to say: this is the same intended action as before. If the previous attempt already completed, return the completed result. If it is still in progress, report that state. If the previous attempt failed before the side effect, allow a safe retry.

That changes what the agent can do. It can retry a narrow operation without guessing whether the world changed. It can recover from a lost response by asking for the state of the idempotency key. It can distinguish “try again” from “do the same external thing again.”

For agents, idempotency is not just an API concern. It belongs in publishing workflows, file edits, task creation, messages, approvals, and any place where duplicate action would break trust.

Structured Outputs Keep the Pipes Fitting

Natural language is a good interface for people. It is a poor interface between every internal component.

A reliable agent system should not ask downstream code to scrape intent out of prose when the shape is known. Tool calls, plans, approvals, validation results, recovery decisions, and final states all benefit from structure.

Structured outputs do not make the model infallible. They make failure easier to detect.

If a field is missing, validation can fail. If a confidence value is outside the allowed range, the run can stop. If a proposed side effect lacks an approval reference, the harness can reject it. If the final state claims a build passed but the evidence field is empty, the acceptance check can catch the mismatch.

The point is not to turn every agent into a form. The point is to reserve prose for the parts that need prose, and use schemas for the parts that need contracts.

A schema is plumbing. It keeps one component's ambiguity from leaking into the next component's assumptions.

Snapshots Make Recovery Possible

Recovery requires knowing what changed.

That is hard if the system never captured the before state.

Before a run mutates something important, it should snapshot the inputs that would be needed to understand or undo the change: the task request, relevant files, tool evidence, configuration, approval state, external object IDs, and the version of the world the agent believed it was operating on.

Snapshots are not only for rollback. They are for explanation.

When a run fails halfway through, the operator should be able to answer: what did the agent see, what did it change, what remains uncertain, and what is safe to resume? When an eval is created from the failure, the snapshot becomes the fixture. When a human reviews the run, the snapshot keeps the review grounded in evidence instead of memory.

Without snapshots, recovery tends to become improvisation.

With snapshots, recovery can become another controlled path through the system.

Backpressure Protects the Operator

Agents can create work faster than teams can safely absorb it.

They can open drafts, generate tickets, run checks, propose fixes, and escalate exceptions. That is useful until every queue is full, every reviewer is blocked, and every failure looks urgent.

Backpressure is how the system says not yet.

Limit concurrent runs. Cap retries. Slow intake when downstream review is saturated. Stop spawning new tasks when the approval queue is full. Prefer fewer complete runs over many half-finished ones. Make the backlog visible instead of hiding it behind optimistic status messages.

This matters because agent reliability is partly social. A system that overwhelms its operators becomes unreliable even if each individual run is technically correct. People stop reviewing carefully. Approvals become rubber stamps. Alerts become noise. Quiet failures get more room to hide.

Good backpressure preserves attention for the work that actually needs judgment.

The Practical Rule

When an agent behaves unreliably, the fix is not always a better prompt or a stronger model.

Sometimes the fix is a queue. Sometimes it is a lock. Sometimes it is an idempotency key. Sometimes it is a schema, a timeout, a snapshot, a dead-letter state, or a clearer approval boundary.

That does not make the model unimportant. It means the model should not be asked to carry responsibilities the system can enforce more directly.

The practical rule is this: make the uncertain part small.

Let the model handle judgment, language, planning, and ambiguity. Let the plumbing handle state, concurrency, durability, validation, and safety boundaries. The more the surrounding system can make explicit, the less the agent has to remember, infer, or hope.

Reliable agents are not built by magic prompts alone.

They are built by giving capable models the boring infrastructure they need to fail safely, recover cleanly, and keep their promises in production.