Agent Observability: Why Traces Are Not Enough

Agent observability is now standard for teams running AI agents: traces, spans, latency, token cost and tool calls. If you run agents without it, start there.

But plenty of teams with excellent observability still can’t get their risk team to sign off. This post explains why, and what to add on top of the stack you already have.

What agent observability gives you

• A trace of every step and tool call
• Latency and cost per run and per step
• Errors, retries and timeouts
• Evaluations of output quality against test sets

Tools like Arize, Phoenix, Langfuse and OpenTelemetry-based stacks do this well, and engineering teams rely on them to keep agents fast, cheap and working.

The question traces can’t answer

A trace shows that an agent called the inventory tool, waited 1.2 seconds, then wrote a maintenance recommendation. It doesn’t show that the tool timed out and the agent assumed the part was in stock anyway.

Observability tells you what the agent did. It doesn’t tell you what the agent believed, or whether your data backed it. That second question is the one risk, compliance and domain experts ask, and it’s why an agent can look healthy on every dashboard and still be wrong.

Observability vs reasoning capture

What to add on top

1. Claims, assumptions and evidence for each decision. AISquare captures these with RML, our Reasoning Markup Language, and checks each claim against your systems. Anything without evidence gets flagged before it becomes a decision.
2. Policy checks recorded per run. Each rule the agent was held to, and whether it passed or failed. Our post on AI agent guardrails covers runtime enforcement in detail.
3. A record that holds up later. A cryptographic signature on every verdict, plus an AI bill of materials listing the models, tools, data and rules used, in the CycloneDX format.
4. Corrections that feed back. Failure clusters, outcomes and reviewed reasoning, kept as precedent so the next run starts smarter instead of from zero.

How the two layers work together

Keep your observability stack. The same agent runs feed both layers, for two different audiences. Engineering uses observability to answer "is it fast, cheap and up?" Risk, compliance and domain experts use the reasoning layer to answer "is it right, is it allowed, and can we prove it?"

You need both answers before an agent earns real autonomy. That’s the core of agentic governance.

A side benefit: lower cost per run

Once reasoning is stored and verified, agents can answer repeat questions from that memory instead of re-deriving everything with a frontier model each time. Model calls on repeat work drop, so cost per run falls as the fleet learns.

Where to start

Pick an agent you already trace. For two weeks, capture its claims and evidence alongside the traces. Then count the runs that looked fine in the trace but carried at least one unbacked claim. That gap is exactly what your risk team is worried about.

Book a demo to see a traced agent run with full reasoning and evidence.

Read next: Agentic governance: how to get agents into production and AI agent guardrails: what they catch and what they miss.