Your agent gave the right answer. You still have no idea how it got there. That is the problem

In February 2025, researchers demonstrated something uncomfortable about large language models.

When they asked a model to solve a problem and explain its reasoning, the explanation the model produced often had almost nothing to do with how it actually reached the answer. The model would give a confident, step-by-step rationale. And that rationale was, in many cases, a post-hoc story. A plausible narrative constructed after the fact, not a faithful account of the computation that produced the result.

The answer was often correct. The explanation was often coherent. And the relationship between the two was far weaker than anyone deploying these systems would like to believe.

This is the reasoning opacity problem. And it sits underneath every governance challenge I have written about in this newsletter so far.

Intent drift, semantic contracts, integration coupling, cost as a control signal. Every one of those is, at some level, a symptom of the same root condition: we deploy autonomous systems that make consequential decisions, and we cannot actually see how they make them. This week I want to name that problem directly, explain why it is harder than it looks, and give you a practical way to govern systems whose reasoning you cannot fully observe.

Why this is not the explainability problem you already know Explainability in machine learning is not new. For years, teams have used techniques like SHAP values and LIME to understand which features drove a model's prediction. For a credit model, you could say: this application was declined primarily because of debt-to-income ratio and recent credit inquiries.

That kind of explainability works because traditional ML models take structured inputs and produce a prediction through a mathematically inspectable process. You can trace the contribution of each feature.

Autonomous agents built on large language models break this. The agent does not take a fixed feature vector and produce a score. It reasons across multiple steps, calls tools, retrieves context, generates intermediate conclusions, and acts on them. Each step involves a model whose internal computation is not decomposable into feature contributions in any meaningful way. Worse, the natural-language reasoning the agent produces, the chain of thought that looks like an explanation, is not a reliable trace of the actual computation. It is generated text. It can be persuasive and wrong at the same time. An agent can produce a flawless-sounding justification for a decision it reached through an entirely different path, and there is currently no robust way to detect the divergence from the reasoning text alone.

This is the trap most organizations fall into. They capture the agent's chain of thought, store it, and believe they have observability. What they have is a plausible story the agent told about itself. Useful, but not the same as knowing what actually happened. The three layers where reasoning becomes opaque Reasoning opacity is not one problem. It manifests at three distinct layers, and each requires a different governance response.

The first is the single-inference layer. Within one call to the model, the actual computation that maps input to output is not observable in a way that maps to human reasoning. You can see the input and the output. You cannot see, in any faithful sense, why this input produced this output. The chain of thought helps but does not resolve this. The second is the multi-step orchestration layer. An agent completing a task makes many inferences, calls tools, and chains intermediate results. Even if each individual step were perfectly transparent, the emergent behavior of the composed sequence is difficult to predict or explain. Small variations early in the chain propagate and amplify. The agent can reach very different outcomes from nearly identical starting conditions, and the reason lies in the interaction between steps, not in any single step. The third is the multi-agent layer. When agents call other agents, the opacity compounds. Agent A's output becomes Agent B's input, but Agent A's reasoning is opaque to Agent B, which treats the output as a given. By the time a decision emerges from a multi-agent system, it is the product of several layers of opaque reasoning interacting in ways no single component can account for. Most enterprise AI governance frameworks address none of these layers directly. They govern inputs and outputs. They assume the reasoning in between is either transparent or unimportant. For autonomous systems making consequential decisions, both assumptions are wrong.

Governing what you cannot fully see Here is the shift in thinking that matters. If you cannot make reasoning fully transparent, you must govern it through behavioral evidence rather than internal inspection.

This is not a compromise. It is how we govern many complex systems we cannot fully inspect, including human decision-makers. We do not have transparent access to how a loan officer reasons internally. We govern their decisions through documented criteria, consistency checks, outcome audits, and escalation paths. The same principles apply to autonomous agents, adapted to their scale and speed.

Four practical mechanisms make this work.

The first is decision boundary testing rather than reasoning inspection. Instead of trying to understand why an agent made a decision, systematically probe what it decides across a wide range of inputs, especially near the boundaries where its behavior should change. If an agent is supposed to approve claims under a threshold and decline above it, test densely around that threshold. You are mapping the agent's actual decision surface, which is observable, rather than its reasoning, which is not. When the decision surface shifts over time, you have detected drift regardless of whether you can explain the reasoning behind it. The second is behavioral consistency monitoring. A trustworthy agent should make the same decision for materially similar cases. Capture decisions in production, cluster similar cases, and monitor whether the agent treats them consistently. Inconsistency on similar inputs is a governance signal that something has changed, even when the reasoning text for each individual decision looks perfectly reasonable. This catches the divergence between plausible explanation and actual behavior. The third is outcome-based evaluation gates. Rather than trusting the agent's stated reasoning, evaluate its decisions against ground truth wherever ground truth eventually becomes available. For a claims agent, the eventual outcome of approved and declined claims is ground truth. For an underwriting agent, actual default behavior is ground truth. Feed this back systematically. An agent whose stated reasoning is impeccable but whose outcomes are degrading is a governance problem the reasoning text would never have revealed. The fourth is reasoning-trace capture with explicit epistemic humility. Capture the chain of thought, but store it and treat it as what it is: the agent's account of itself, not a verified trace. It is useful for investigation, for spotting patterns, for generating hypotheses about what went wrong. It is not proof of how the agent reasoned. Governance frameworks that treat captured reasoning as ground truth are building on sand. Ones that treat it as one signal among several, to be corroborated against behavioral evidence, are building on something more solid. Why this matters more as agents gain autonomy The reasoning opacity problem is manageable today largely because most enterprise agents operate with a human in or near the loop for consequential decisions. The human provides a check that compensates for the opacity.

As organizations move toward greater autonomy, removing the human from more decision loops to gain speed and scale, the compensating check disappears. The opacity remains. And the gap between what the agent appears to be doing, based on its reasoning text, and what it is actually doing, based on its behavior, becomes a live risk rather than a theoretical one.

The organizations that will deploy autonomous AI safely are not the ones waiting for perfect explainability. That may never come. They are the ones building behavioral governance now, so that when they remove the human check, they have something rigorous in its place. This is the discipline that connects every issue I have written in this newsletter. Intent drift is detected through behavioral monitoring. Semantic drift is caught through decision boundary testing. Cost anomalies are a behavioral signal. All of it is governance through observable behavior rather than through internal inspection, because internal inspection of these systems is not reliably available.

Your 30-day and 90-day plan 30 days: Map your reasoning-opacity exposure Identify every autonomous agent making consequential decisions in your organization. For each one, answer honestly: do we currently rely on the agent's stated reasoning as evidence that it is behaving correctly? If the reasoning text and the actual behavior diverged, would we know?

For your most consequential agent, establish a decision boundary test. Identify the thresholds where its behavior should change. Construct a test set that probes densely around those boundaries. Run it. Document the current decision surface as a baseline.

For most organizations, this exercise reveals that governance has been relying on plausible reasoning text far more than anyone realized, and that no baseline of actual decision behavior exists.

90 days: Build behavioral governance For your highest-consequence agents, implement the four mechanisms as a working system.

A decision boundary test suite that runs regularly against production behavior and alerts on shifts in the decision surface.

Behavioral consistency monitoring that clusters similar production cases and flags inconsistent treatment.

An outcome-based evaluation loop that feeds ground truth back and compares stated reasoning against actual results.

Reasoning-trace capture stored with explicit governance status: useful for investigation, never treated as verified proof of computation.

At 90 days, your governance of these agents rests on observable behavior corroborated across multiple signals, not on the stories the agents tell about themselves.

The question for your next governance review For your most consequential autonomous agent, if its stated reasoning stayed perfectly coherent while its actual decisions quietly drifted, how long would it take you to notice?

If the honest answer is that you would not notice, you are governing a story, not a system.

Next issue We have now diagnosed six problems across this newsletter. Intent drift. Multi-agent governance. Semantic contracts. Cost as a control signal. And now reasoning opacity.

Next week is the pivot. Because these are not six problems. They are six symptoms of one architectural condition. And once you see the single problem underneath them, the entire approach to governing enterprise AI changes.

AI Pulse · Issue 06 (The Pivot) One Architecture Problem: Why Everything You Have Been Fighting Is the Same Fight

Until Next Week

Rupali