An agent that can act in a business system needs a record of how it got there.
Andrej Karpathy once called English “the hottest new programming language.” That shift makes software creation more accessible. It also means more business logic is being expressed through prompts, tool permissions, retrieval rules, and model behavior that can change between runs.
When an agent takes an action, a polished final answer is not enough. The organization needs to reconstruct what the agent saw, what it decided, which tools it called, what came back, and what changed in the system.
A final answer is not an audit trail
Consider an agent that updates a customer record after reading an email and checking an order system. If the update is wrong, “the model made a mistake” does not tell an operations leader what to fix.
Was the email misleading? Was the source record stale? Did retrieval return the wrong policy? Did the agent call the wrong tool, or did the tool return an unexpected result? Was the action within the employee’s authority? Did a retry duplicate the update?
Those are different failure modes. They require different owners and remedies. A useful record connects the user request, relevant evidence, model and prompt version, tool inputs and outputs, policy checks, approvals, retries, and resulting system changes. Sensitive data should be redacted or access-controlled, but the decision path still needs to be inspectable.
Instrument the work, not just the model call
agentevals is an open-source example of evaluating agent behavior from OpenTelemetry traces. It can score recorded sessions against evaluation sets without rerunning the agent, which makes it a useful reference for thinking about how traces can support repeatable evaluation.
The implementation lesson is broader than any one repository. Treat an agent run as a trace across the workflow. Give each consequential action a stable identity. Record the evidence and policy outcome that authorized it. Preserve tool results and the final state change. Then connect incidents and near misses back to evaluations that can run in CI or before a release.
That creates a feedback loop. Operations can investigate a case. Product can see which behavior is recurring. Engineering can reproduce the failure. Risk and compliance can inspect the controls without relying on someone’s memory of a demo.
Make observability part of the business case
The question for a CIO is not simply whether the agent has logs. It is whether the organization can explain and control its behavior at the point where it affects customers, money, or records.
The CFO should care too. If the system cannot explain why it took an action, the cost of a mistake includes investigation, remediation, customer impact, and lost confidence. Observability also shows whether the agent is completing work or creating hidden review and rework.
I would make a reconstructable decision trail a release requirement for any agent with meaningful permissions. The model can be impressive. The business system still has to be accountable.
Leave a Reply