When Tool Calls Succeed but Actions Fail: Lessons from AI Governance

It was a textbook debugging session. The logs showed everything working: refund_order had been called, the order ID matched, and the API returned success. But the user had only asked whether the order could be refunded—not to actually process the refund. The tool call succeeded, but the business action was wrong.

This scenario, drawn from a series on Knowledge-driven Computing (KDC), highlights a growing problem in AI operations. As agents move from answering questions to taking actions, the old ways of logging and auditing no longer suffice. A successful API call doesn't mean the agent made a correct or authorized decision.

The Limits of Traditional Logging

In traditional software, a refund request follows a predictable path. The user clicks a button, confirms a second time, the backend validates order status and permissions, risk rules decide whether human review is needed, and the transaction service executes the refund with retry and rollback logic. Developers predefine when to call, who can call, what to check before calling, and what to do on failure.

Thus, even a simple log entry—interface, parameters, result—can be traced back to the code version, workflow, and rules that governed the action. The causal chain is embedded in the program.

But agent systems change this. The model interprets the user's goal at runtime, combines context, selects tools, and decides whether to act. The same phrase, "Can I return this?" could mean a rules inquiry, a refund intention, or a comparison of options. The choice is no longer fully determined by pre-written code paths.

When important actions are chosen at runtime, the reasons for those actions need to become manageable engineering objects. Otherwise, the system gains more and more execution capability, but explains its actions with poorer and poorer logs.

Introducing the Reasoning Object

KDC proposes that high-impact judgments leave behind an external, auditable structure called a Reasoning Object.

It's not a prompt. A prompt organizes the model's input, but it doesn't record what evidence was used, what conclusions were drawn, or what risks were assessed.

It's not a trace. Traces track which components were hit and how long they took, but they miss the business semantics and causal basis of the judgment.

It's not a full chain-of-thought. The model's internal reasoning may be incomplete, unstable, or not suitable for exposure. KDC cares about the externally auditable structure, not the verbatim internal monologue.

For a high-impact judgment, the Reasoning Object should capture:

• Goal: What the task was supposed to answer or accomplish
• Context: User, order, time, session, and system state
• Knowledge references: Which knowledge objects and versions were used
• Memory references: Historical preferences, experiences, or precedents that influenced the judgment
• Evidence: Facts, documents, states, or human confirmations that support the conclusion
• Key judgments: The externally checkable steps from goal to conclusion
• Conclusion type: Factual, recommendation, plan, or action request
• Uncertainty: Whether evidence was sufficient or more info needed
• Risk: Potential impact if the conclusion or action is wrong
• Action recommendation: Whether to invoke a capability
• Feedback: Whether the real-world result validated or refuted the original judgment

In the refund case, a proper structure would separate two conclusions: "the order meets refund conditions" (a factual judgment) and "the user explicitly requested an immediate refund" (an authorization judgment). These are distinct, and the system must not merge them because they sound similar in natural language.

From Judgment to Action: The Causal Anchor

An AI action should leave behind more than a tool call record. It should form a chain: business goal → knowledge and evidence → reasoning object → capability selection → policy and permission check → capability invocation → execution result → real-world feedback.

The reasoning object sits between knowledge and action, serving as the upstream causal anchor. It tells auditors that the refund wasn't because the model "felt like" calling a tool, but because the system recognized a goal, referenced the current refund policy, read the order facts, assessed risk, and formed a recommendation.

It also makes refusal a normal outcome. If evidence is insufficient, the goal is ambiguous, the user hasn't confirmed, or knowledge conflicts, the reasoning object can explicitly state "cannot act now." The governance system doesn't have to wait for the tool to fail to discover the premise was false.

Not every action needs a full reasoning object. Low-risk read-only queries can skip it. But refunds, payments, permission changes, external notifications, approval submissions, and irreversible operations should at least retain equivalent records of goal, evidence, risk, and authorization.

The Reasoning Runtime: Managing Judgments, Not Just Prompts

The reasoning object defines what a judgment should look like. The Reasoning Runtime is the system component that creates, uses, and updates those objects during operation.

It handles tasks like identifying the goal type, assembling context from knowledge and memory, binding evidence, creating the reasoning object, expressing uncertainty and risk, proposing capability choices, and linking feedback back to the original judgment.

The model can be part of this, but the runtime isn't the model itself. Goal classification can be done by rules and models together; evidence completeness can be checked by the system; risk level can be set by policy; user confirmation can come from interaction state; feedback may arrive long after the model call ends.

The runtime isn't a prompt orchestrator either. A prompt organizes a single model input, but the runtime manages knowledge versions, object identity, risk status, action recommendations, and follow-up feedback.

In early systems, these responsibilities don't need to be deployed as a separate platform. They can be handled by application modules, structured records, and policy checks. The point is to have a clear owner for the judgment lifecycle, not to create a new service called "Reasoning Runtime" prematurely.

Tool, Skill, Capability: Three Levels of Execution

Once the reasoning runtime has formed a judgment, the system still needs to decide how to execute it. This is where distinguishing Tool, Skill, and Capability becomes critical.

Tool is the minimal executable entry point. It answers "how to execute a specific operation." For example, get_order_status queries order status, check_refund_eligibility checks refund conditions, refund_order initiates the refund, and send_notification sends a notification. A tool describes its name, parameters, and return values. But just because a tool can be called doesn't mean it's appropriate for the model to call in every context.

Skill organizes multiple tools around a business goal. "Handle order refund" isn't a single tool; it might include goal clarification, order lookup, policy retrieval, eligibility check, user confirmation, refund execution, status tracking, and escalation to human support. A reusable skill should express the business goal, preconditions, required context, participating capabilities, orchestration logic, risk boundaries, failure and degradation strategies, and audit requirements. Skills sit between fixed workflows and fully free-form planning—they don't hard-code every step, but they also don't let the agent decide everything from scratch each time.

Capability is the governance-aware execution abstraction. It answers "under what conditions can this action be understood, selected, called, and governed?" A capability like "initiate order refund" includes more than the underlying tool: it has an identity, business semantics, input/output effects on real-world state, calling subject and permissions, risk level, owner and responsibility domain, version and lifecycle, preconditions and post-invariants, human confirmation requirements, audit and observability policies, and rollback or compensation methods.

In short: Tool = how to execute; Skill = how to organize execution around a goal; Capability = how to execute within governance boundaries.

Why Prompt Constraints Aren't Enough

Teams often write rules like "must obtain user confirmation before executing a refund" in the system prompt. That's a useful instruction, but it shouldn't be the only defense for high-risk actions.

Prompts are affected by context, models can misinterpret natural language, and instructions can be lost in complex tasks. If the underlying refund_order tool is fully open to the model, governance responsibility is effectively compressed into a single probabilistic judgment.

Capability governance needs to elevate key boundaries to system mechanisms:

• Permissions: Does the current subject have the right to initiate a refund?
• Risk level: Does the amount, order status, or scenario require stricter controls?
• Policy decisions: Are knowledge, evidence, and preconditions complete?
• Human-in-the-loop: Is user confirmation or human approval required?
• Rate limiting and circuit breakers: Should abnormal calls be suppressed or halted?
• Transactions and compensation: How to avoid half-completed states on failure?
• Audit: Who initiated the action based on what judgment?
• Observability: What are the success rates, policy rejections, human interventions, and real-world results?
• Lifecycle: Is the capability in experimental, grayscale, stable, or retired status?

In the refund case, the control plane could require an explicit user confirmation as a precondition for the transactional refund capability. Even if the agent reasons incorrectly or the skill orchestration misses a step, the capability won't execute.

The Agent as Coordinator, Not Container of All Responsibilities

If you treat the agent as an all-powerful entity, it ends up responsible for understanding goals, storing all context, forming judgments, selecting tools, executing actions, handling failures, and generating audit trails. The system looks simple, but responsibilities can't be separated.

When something goes wrong, you're left with "the agent made a mistake." But the real cause could be in goal interpretation, knowledge version, memory usage, reasoning path, skill selection, capability implementation, governance policy, or the external system.

KDC therefore doesn't start with the agent. It places the agent on top of objects, runtimes, and governance mechanisms. The agent can interpret user goals, request and organize context, coordinate reasoning, select reusable skills, propose capability calls, and handle execution feedback. But high-impact judgments should form reasoning objects or equivalent audit evidence, and high-risk actions should go through capability governance. The agent can suggest an action, but just because it reasoned something doesn't give it automatic permission to execute.

This also avoids the opposite extreme of turning the agent into a fixed script. Goal interpretation and path selection can still be dynamic, but key responsibilities aren't hidden in the agent's temporary context.

Integrating with Existing Infrastructure

KDC doesn't require reinventing model connection protocols, agent frameworks, identity systems, or gateways. MCP and tool-use solve how models discover and invoke resources. Agent frameworks handle goal decomposition, task execution, and multi-agent collaboration. API gateways manage authentication, routing, rate limiting, and traffic governance. IAM and policy engines handle subjects, resources, actions, and policy decisions. Workflow and approval systems carry stable processes and human nodes.

The question KDC adds is how to connect these to a single business causal chain: What business capability does an MCP tool correspond to? Why did the agent recommend calling it? What knowledge did the reasoning reference? What is the current risk? What context did the policy engine base its decision on? And how does the execution result feed back to the real world?

The capability control plane is a theoretical abstraction of these governance responsibilities, not a specific product. It can be implemented with existing API gateways, IAM, policy engines, risk control, approval, and observability systems.

Practice: Reconstructing a High-Risk AI Action Audit Chain

Pick a high-impact action in your system—refund, payment confirmation, permission change, approval submission, order creation, or external notification. Don't design a new platform first. Take a real call record and try to reconstruct the chain: business goal → knowledge and memory evidence → auditable judgment and risk → skill and capability selection → permission and policy check → user confirmation or HITL → execution result → real-world feedback.

Then create a minimal AI Action Record. Look for four types of gaps:

• Causal gap: You see the tool call but can't trace the goal and evidence
• Semantic gap: The tool has parameters but no business meaning, risk, or owner
• Governance gap: Key boundaries exist only in the prompt, not in system policy
• Feedback gap: The API returned success but there's no verification of the real outcome

This record doesn't need to be uniform across all systems, and not every read-only query needs a full structure. Its purpose is to verify: For important actions, can the system answer "why was this done, who allowed it, based on what evidence, and what was the result?"

If the answers still require manually piecing together chat logs, prompts, and multiple log files, then your action audit chain isn't a stable engineering capability yet.

The Real Challenge: Knowing When Not to Act

Getting a model to call a tool is no longer the hard part. The hard part is linking runtime judgment, business risk, and execution permissions into a single governable chain.

Reasoning objects let the system express "why this judgment." Skills make goal-level workflows reusable while preserving dynamic space. Capabilities give actions semantics, permissions, risk, owner, and lifecycle. Control planes ensure key boundaries don't rely solely on model self-discipline.

So remember: A successful call doesn't mean the judgment was correct. A correct judgment doesn't mean the action was authorized. An API returning success doesn't mean the real-world goal was achieved.

The value of enterprise AI isn't in having the agent complete as many actions as possible. It's in acting when it should, stopping when uncertain, unauthorized, or under-evidenced, and making every important action explainable, auditable, and accountable.

But even with perfect reasoning and execution, the system isn't reliable long-term. Reality changes, knowledge expires, users correct the system, and capabilities fail. The next step is understanding how history and feedback become memory—and how to avoid amplifying mistakes. That's a story for another day.