The Agent Harness: Where Reliability Actually Lives

An LLM is a function from text to text. It has no memory of your systems, no ability to act on them, and no way to know whether what it just produced is true. Everything that makes an agent useful in production sits around the model rather than inside it, and that surrounding layer is the harness.

Teams consistently underestimate it. They spend their evaluation effort on models, then discover that swapping a good model for a better one moved their success rate by a few points while a fairly ordinary fix to retrieval moved it by thirty.

Whatever framework you pick, they all converge on the same loop. Something assembles context. The model proposes an action. Something decides whether that action is permitted. Something executes it. Something checks the result. If the check fails, round again.

That shape is right as far as it goes, and it is worth being able to draw from memory. It is also missing four things that determine whether the system survives contact with production. Because the missing pieces are unglamorous infrastructure rather than model behaviour, they tend to get built last, normally in the week after an incident.

The loop, in one pass

Worth agreeing on the baseline before adding to it.

Context assembly turns a goal into a prompt using retrieval, prior turns, and whatever state the task has accumulated. The model reasons over that and proposes one action, usually a tool call. A gate decides whether the action is permitted. A runtime executes it and hands back the result. Verification decides whether that result is acceptable, and on failure puts the reason back into context for the next pass. When verification passes, the result goes to the user. Running alongside all of it are telemetry, so you can see what happened, and constraints, so the gate has something concrete to enforce.

Now read that back and notice the questions it does not answer. How many times does the loop run before you give up? What happens when the gate says no and there is no permitted alternative? Where does memory live between sessions rather than within one? And what stops a passing verification from simply being a badly written test?

What the standard loop leaves out

The first gap is a loop budget. That cycle has no bound on it, and in production the expensive failure is not a wrong answer, it is an agent that calls a tool, fails verification, retries, and burns four hundred model calls before anybody notices. You need a hard ceiling on iterations, tokens, wall-clock time and money, enforced by the harness rather than hoped for.

The second is memory as a distinct tier. Lumping documents, data and history into one box collapses three things that have different lifetimes. Working memory belongs to this task. Long-term memory survives across sessions. Retrieved knowledge is neither of those, it is a lookup. Different write paths, different eviction rules, different ways of going wrong.

The third is explicit terminal states. The usual diagram has exactly one exit, which is success. A real harness needs at least two more. Refused, when the gate blocks and no alternative exists. Exhausted, when the budget runs out mid-task. Both have to be first-class outcomes the caller can handle rather than exceptions falling out of a loop.

The fourth is tool result caching. Agents re-ask the same questions constantly, and they do it most when looping after a failed verification. Without a cache keyed on the tool call you pay full latency and full cost for an answer you had thirty seconds ago.

The redesigned architecture

Context assembly

This is four jobs, not one. Retrieve candidates, rank them, compress what survives, and pin what must never be dropped.

The pinning matters more than people expect. System instructions, the user's actual goal and any safety-relevant facts have to be immune from the compression step. Otherwise a long conversation quietly evicts the instruction that was keeping the agent honest, and the resulting failure looks like the model forgetting something rather than the harness throwing it away.

Things go wrong here through stale retrieval, wrong ranking, and truncation of something load-bearing. There is also a subtler failure, which is retrieving too much and burying the relevant fact in the middle of the context where models attend to it least.

Instrument the number of chunks retrieved against the number actually used, token counts split by category so you can see instructions against retrieved content against history, and how often compression fires.

The model call

One rule is worth enforcing structurally: the model proposes and never executes. Tool calls come back as data, not as side effects.

This sounds obvious and gets violated routinely by frameworks that let model output invoke code directly. Doing so collapses the propose, gate and execute stages into one and removes your ability to say no, which is the entire point of having a gate.

Failures here look like malformed tool arguments, hallucinated tool names, and confident reasoning over context that did not contain the answer. The metric to watch is the schema-validation failure rate on proposed calls, because a rising rate usually means your tool descriptions have drifted away from what the tools actually do.

The policy gate

Three outcomes rather than two. Allow, block, and require human approval. That third one is what makes an agent deployable against anything that spends money or touches customer data.

The gate also has to reason about the class of an action rather than its name. Sending an email to an internal address and sending one to a customer list are the same tool and entirely different risks, which means the gate reads arguments and not just the function signature.

Rules go wrong by being too broad, in which case the agent cannot do its job, or too narrow, in which case something gets through. The third failure is a rule that cannot be evaluated at all because the argument determining risk is buried inside a free-text field.

Track block rate by rule, and approval latency. If humans are approving ninety-nine percent of requests without reading them, the gate is theatre and you should know that before an auditor tells you.

The tool runtime

The runtime's job is making execution survivable. Timeouts on everything, idempotency keys on anything mutating, retries only where they are safe, and hard separation between read-only and state-changing calls.

That separation earns its keep in two specific places. The tool cache can serve read-only results freely and must never serve a mutation. And retry logic can be aggressive on reads while staying conservative on writes.

The failures are a write executed twice after a retry, a tool that hangs without a timeout and eats the entire loop budget, and error messages so unhelpful that the model cannot recover from them.

That last one is badly underrated. Error: 400 teaches the model nothing at all, whereas Error: end_date must be after start_date; you sent 2024-01-05 and 2024-01-01 lets it fix itself on the next iteration. Tool error messages are part of your prompt engineering, whether you treat them that way or not.

Instrument per-tool latency and error rate, retry counts, and cache hit rate.

Verification

The stage that decides whether any of the rest mattered, and the one most often implemented as a vibe check.

Real verification is layered, cheapest first. Schema and type checks tell you whether the shape is right. Assertions derived from the goal tell you whether the content is right, so if the task was to book a flight under ₹40,000 then something asserts the price. That is the layer people skip, and skipping it is most of why agents fail silently. Cross-checks against a source come next, meaning you re-read the record you just wrote. A second model as judge belongs last, because it is both the most expensive and the least reliable of the four.

What goes wrong: tests that pass on wrong output, tests that check form rather than substance, and feedback so vague that the next iteration repeats the same mistake.

Instrument pass rate by check type and how many iterations verification adds. A check that never fails is not a check, it is decoration.

The three terminal states

Success is the easy one. The other two have to be designed, because they are what your caller sees on a bad day.

Refused means the gate blocked and there was no permitted alternative. The response needs to say what was blocked and which rule blocked it, so a human can work out whether the rule or the request was the thing that was wrong.

Exhausted means the budget ran out, and it must return partial progress rather than nothing. An agent that spent ₹200 and thirty seconds and then reports failure has thrown away work somebody paid for. Return what was accomplished, what remains, and where it stopped.

Where failures actually come from

The question that usually follows this diagram is which layer breaks most. My answer has two halves, and I should be clear that the numbers below are my judgement from watching a number of these systems rather than measured data from a study.

Most failures originate in context, somewhere in the region of half to sixty percent. The model reasoned correctly over a context that was stale, incomplete, or so bloated that the one relevant fact was effectively invisible. It is also the least interesting layer to work on, which is precisely why it stays broken while people tune prompts.

The failures that hurt, though, originate in verification, or rather in its absence. A context failure with good verification is a retry. The same context failure without it is a wrong answer delivered with total confidence. Verification does not prevent many failures so much as convert silent ones into visible ones, which turns out to be worth more.

Layer	Share of failures	Character
Context assembly	~50–60%	Silent. Looks like the model being dumb.
Verification	~15–20%	Passing tests that should not pass.
Tools and runtime	~15%	Loud, and therefore mostly already fixed.
Constraints and gate	~5–10%	Rare, expensive when wrong.

Tools fail most often and matter least, because tool failures are loud. Something throws, something returns a 500, somebody gets paged. Context failures are quiet, and the quiet ones are what reach users.

The data layer nobody draws

Every dashed box in the side column of that diagram is a data-layer problem, and this is the point where agent architectures quietly turn into database architectures.

Memory tiers have different physics. Working memory is small and hot, read every single turn, discarded when the task ends, which describes a keyspace with a TTL. Long-term memory is written rarely, retrieved semantically, and has to survive indefinitely, which describes a vector index. Treating them as one store means either paying vector-search latency for something you should have read by key, or losing state you needed next week.

Semantic caching pays for itself inside the loop. When an agent retries after a failed verification it re-asks near-identical questions, which an exact-match cache misses and a semantic cache catches. Same mechanism as user-facing Q&A, except the repetition rate inside a retry loop is far higher than it ever is across users.

The tool cache is where your latency budget is won or lost. A five-step loop with four tool calls at 200ms each has spent 800ms before the model has thought about anything. Cache the read-only calls and that becomes 800µs on a hit.

One number worth carrying around: a single agent turn with one context retrieval, one model call and two tool calls is four round trips, multiplied by your step count. Agents make orders of magnitude more data requests than human users do, which is why retrieval infrastructure that was sized for human-paced traffic starts buckling the first time you point an agent at it.

The loop in code

Stripped to the bones, so the budget and the terminal states are visible:

def run(goal, budget):
    ctx = Context(goal)                       # pinned instructions + goal
    while budget.remaining():
        prompt = ctx.assemble()               # retrieve, rank, compress, pin
        action = model.propose(prompt)        # proposes only
        budget.charge(action.tokens)

        decision = gate.evaluate(action)      # reads args, not just the name
        if decision.blocked:
            return Refused(action, decision.rule)
        if decision.needs_approval:
            if not approvals.request(action):
                return Refused(action, "human declined")

        result = runtime.execute(action)      # timeout + idempotency key
        ctx.add_observation(result)

        check = verify(goal, result)          # schema, assertions, cross-check
        if check.passed:
            return Accepted(result)
        ctx.add_feedback(check.reason)        # specific, actionable

    return Exhausted(ctx.progress_so_far())   # never return nothing

Four properties that the structure enforces rather than requesting politely: the model cannot execute, the gate cannot be bypassed, the loop cannot run forever, and every exit path returns something the caller can act on.

If you are building one

The order of investment I would argue for follows the failure distribution. Context assembly first, because that is where most failures start. Verification second, because it is what makes the remaining failures visible instead of silent. Bound the loop third, before it costs you real money. Tune the model after all of that.

Instrument from day one, though. The most useful artefact in an agent system is not the prompt, it is a trace of one complete loop showing what was retrieved, what was proposed, what was allowed, what came back, and why verification passed or failed. Without that you are guessing, and with agents the guesses have a bill attached.