The Curse of Non-Determinism: Putting agentic AI into production

The pattern repeats almost every time we walk into a client engagement. The pilot looked great. The model was accurate, the retrieval was clean, the demo landed. Then the system went into production and something began to go wrong that nobody could quite name. The same question started returning different answers depending on who asked it. The same ticket got escalated on Tuesday and resolved silently on Wednesday. The same contract clause got summarized one way for the legal team and another for sales. Nothing was wrong, exactly. But nothing was the same twice.

The instinct in this situation is to ask whether the model is accurate enough. It's the wrong question. Accuracy is a property of predictions; production AI systems don't make predictions, they enact policies, they retrieve, they reason, they call tools, they remember. The right question, the one we've come to ask first in every engagement, is about behavior: not whether the system gets a single answer right, but how its behavior is distributed across runs, prompts, contexts, and time.

We've come to call this the curse of non-determinism, and we think it's the central engineering problem of agentic AI in production.

Your AI agent doesn't have an accuracy problem. It has a variance problem, and accuracy metrics are structurally unable to see it.

There's a precedent worth naming. Two decades ago, machine learning ran into the curse of dimensionality: as you add features to a model, the feature space grows exponentially relative to your data, sample sparsity destroys distance metrics, and generalization quietly collapses. The lesson the field eventually internalized was structural, don't add features blindly. Constrain the space through regularization, feature selection, and inductive bias.

Agentic AI has its own version of this problem. Every degree of freedom we add to an agent whether it's a new tool, a longer context window, a memory store, a retrieval layer, or an additional model compounds the behavioral space the system can occupy. The math is different; this is not a statistical estimation problem. But the structural lesson rhymes. More freedom without architecture is not more capability. It is more variance.

The discipline this requires doesn't yet have a name in most organizations. It should. We call the thing being managed behavioral variance, and the rest of this piece is about how to measure it, where it comes from, and what it costs to govern.

For a decade, enterprise machine learning was about classifiers. The output space was small, ground truth was assemblable, and the natural metric was accuracy. Generative AI broke this regime in three ways at once.

The output space became open. There is no enumerable list of acceptable answers to "summarize this contract." The input became contextual. Retrieved documents, prior conversation, and tool results mean the same nominal prompt rarely produces the same effective input. And the system started acting. A tool call is not a prediction; it's an event in the world, often with side effects, sometimes irreversible.

What we need to characterize, in this regime, is not a prediction but a policy, a mapping from situations to distributions over what the system says and does. Two policies can have identical pointwise accuracy on a held-out evaluation set and behave very differently in production. One drifts gracefully when a document is updated; the other doesn't. One refuses cleanly when it's outside its competence; the other confabulates. One produces consistent answers to paraphrased questions; the other doesn't.

Two systems with the same accuracy can have entirely different behavioural envelopes. Accuracy cannot tell them apart. Variance can.

The first useful move is to stop talking about non-determinism as if it were one thing. In production systems we see at least five mechanistically distinct sources of variance, and confusing them is the most common diagnostic error we encounter.

These sources interact. A small prompt change alters the rewritten retrieval query, which alters the retrieved context, which alters tool selection, which alters the memory written for the next turn.

When something goes wrong in production, the question isn't was the model right, it's which layer's variance produced this outcome.

Most teams can't answer because their observability collapses all five into a single trace of "what the model said."

Calling for a "variance budget" without saying how to measure variance is the kind of governance theater the AI field already has too much of. The measurement question has to be operational.

For tasks with structured outputs such as classifications, extractions, code, the natural object is the agreement rate across N runs: the fraction of runs that produce the modal output, or the entropy of the empirical output distribution. For free-form generation, where agreement is ill-defined, we fall back on pairwise semantic similarity across runs, with the caveat that these metrics are themselves noisy. Track them as trends on a fixed evaluation set rather than as absolute values. For tasks with verifiable outcomes, a tool call that either succeeds or fails, a number with a known range, measure outcome variance directly. This is the most informative metric, because it integrates over all the intermediate variation that doesn't actually affect what happened.

A note on LLM-as-judge, since it is now everywhere. Using a strong LLM to grade another LLM's outputs is useful, but it has known biases: judges prefer the first answer presented, prefer longer answers, prefer outputs from their own model family. These biases don't invalidate the technique, but they require that judges be calibrated against human review on a sample, that prompts mitigate known biases (randomized order, length-controlled comparisons), and that the judge itself be evaluated for variance. Treating an LLM judge as ground truth is a category error.

A system that knows when it doesn't know is governable. One that doesn't, isn't, and no amount of accuracy compensates.

A genre of AI commentary now claims that agentic systems require an entirely new enterprise architecture. Some of this is true. Much of it is the rediscovery of separation of concerns under a new label. The intellectually honest move is to name which is which.

What is not new: keeping commitments, financial limits, identity, and audit in deterministic systems while a probabilistic component handles language and presentation. This pattern is decades old. Every chatbot worth its salt in 2005 routed authoritative actions through a rules engine. Calling this "controlled autonomy" rebrands sound design as innovation.

What is new is the surface area, in three respects.

First, the capability surface scales super-linearly with tools. A traditional system exposes a fixed operation set, each invoked through known code paths by known callers. An agent with k tools can compose them in vastly more trajectories of length n, and the choice among trajectories is made by a probabilistic policy whose decision boundary cannot be statically inspected. This is why "approve the application" is no longer the right unit of architectural review. The unit is the capability: what can be invoked, by whom, with what parameters, with what side effects. The old idea of capability-based security, dating to the 1960s, turns out to be the right primitive, but its application to LLM agents is genuinely new and largely unsolved.

Second, the input is no longer trusted. Classical systems treat user input as untrusted but treat surrounding context configuration, retrieved data, tool results as trusted. LLM agents have to treat all of it as adversarial. The phenomenon now called indirect prompt injection demonstrates that an attacker who controls a document the agent retrieves, or a webpage it reads, can cause the agent to execute attacker-chosen actions. There is no classical analog for a system whose control flow can be hijacked by the data it processes. This isn't a governance problem solvable by policy; it's an open security problem.

Third, behavior depends on the corpus. In RAG systems, the effective program is the union of the prompt, the model weights, the retrieval index, and the documents in the corpus at query time. Change any of these and behavior changes. The corpus is typically managed by a different team than the application, on a different cadence. The architectural implication is that the document lifecycle becomes part of the application lifecycle. Documents need versioning, freshness SLAs, classification, and rollback, treated with the same seriousness as code. Most organizations are not ready for this, because most organizations had not solved enterprise document governance before AI made it urgent.

The right framing is therefore neither "this is just classical architecture" nor "everything is new." Determinism around commitments is old. Capability-bounded agents, prompt-injection defense, and corpus-as-code are new. The literature is still catching up.

The governance prescriptions popular in enterprise AI commentary, comprehensive observability, retrieval governance, tool permissioning, human-in-the-loop, continuous evaluation, are not free. We've watched these costs blow up budgets often enough to think the field needs to talk about them honestly.

Evaluation has compute cost. Running a meaningful regression suite per prompt change, against multiple model versions, with self-consistency sampling, can dominate the model bill itself. Teams that propose continuous evaluation rarely budget for it.

Human-in-the-loop has throughput cost. A human reviewer at €50 per hour reviewing thirty-second tasks gives an effective floor of roughly €0.40 per task in labor. For a system handling 100,000 tasks a day, even five percent sampling costs €2,000 daily. For low-margin workflows this can defeat the use case. The alternative, review only low-confidence outputs, depends on the calibration property that, as we noted, is rarely satisfied off-the-shelf.

Observability has storage and engineering cost that lacks a corresponding visible feature, which is precisely why teams underestimate it. Retrieval governance has organizational cost: source classification, freshness monitoring, citation policy, and access-aware retrieval are operating procedures requiring humans to maintain them, not features to configure.

The honest implication is that some workflows shouldn't run on LLM-based systems. If the task volume is too low to amortize evaluation, the consequences are too high to tolerate residual variance, and the upside is incremental rather than transformational, the variance budget cannot be made to pencil out.

This is not a failure of AI strategy. It is the framework working as intended.

Its job is to make this conclusion legible, not to greenlight every use case.

A useful diagnostic: compute the ratio of value per task to governance cost per task. When this ratio is below one, the use case fails the budget regardless of how good the model is. We've come to insist on this calculation before any client engagement that involves moving an agent into production. It changes which projects get built and which ones get redesigned.

If we had to compress everything we've learned into the first questions we ask before scaling an agentic system, they would be these.

Variance is not the enemy. Variance is often where AI creates value: a useful sales agent adapts, a useful analyst surfaces a non-obvious framing, a useful coding assistant explores. The goal is not deterministic AI. The goal is to make the envelope of behavior, the region within which adaptation is allowed, explicit, observable, and accountable, while keeping commitments and irreversible actions outside that envelope.

The companies that build durable AI capability won't be the ones that adopt fastest or the ones that refuse longest. They'll be the ones that learn to specify the envelope precisely, measure when the system leaves it, and pay the cost of governance where the value justifies it and walk away where it doesn't.

That, in our experience, is the actual work.