LLM Agent Observability: Trace Every Step of Your Agent

An LLM agent is a loop: the model decides, calls a tool, observes the result, decides again, and repeats until it produces an answer. That loop is the hardest thing you will ever debug. When an agent gives a wrong answer, the bug can live anywhere — in the prompt, the tool implementation, the tool result parsing, the loop termination logic, a rate limit, or a hallucinated intermediate step — and the answer itself gives you no clue which. Ordinary API monitoring tells you the loop ran; agent observability tells you what the agent actually did: every model call, every tool invocation, every token, every decision, in a replayable trace. This guide covers the structure of agent traces, OpenTelemetry's GenAI semantic conventions, cost tracking per trace, evaluation integration, and the debugging workflows that make agent systems maintainable.

Why Agents Are Harder to Observe Than APIs

Classic API observability is built around a single request: one span, one status, one latency. An agent is a tree of requests — each model call may trigger tool calls, each tool call produces results that feed the next model call, and the whole tree shares one user-visible outcome. Three properties make this genuinely different:

• Branching and loops. Agents can run 2 or 20 model calls for the same user question. Without tracing, you cannot even tell how many steps ran, let alone which one went wrong.
• Side effects outside your stack. A tool call hits your database, an external API, or a filesystem. Failures and partial writes happen in tools, not in the model — and the model may not notice.
• Non-deterministic reasoning. The same input can produce different tool sequences on different runs. A bug that reproduces once in ten runs is invisible to logs and needs traces to diagnose.

There is also a cost dimension: an agent run can spend $0.01 or $5.00 depending on how many steps it takes. If you cannot see the steps, you cannot control the cost. Observability is what makes agents shippable — it is the difference between "the agent is flaky" and "the agent failed at step 3 because the weather tool returned an empty payload."

The Anatomy of an Agent Trace

A trace is a tree of spans. For an agent run, the canonical structure looks like this:

trace: "customer-support-agent run #4821"
├── span: agent-loop (root)
│   ├── span: llm-call (step 1: decide)
│   │   ├── attributes: model, input tokens, output tokens, latency
│   │   └── events: tool_choice = get_order_status
│   ├── span: tool-call get_order_status
│   │   ├── attributes: tool name, input args
│   │   ├── events: http request to order-service (200, 32ms)
│   │   └── output: JSON payload
│   ├── span: llm-call (step 2: respond with data)
│   │   └── attributes: tokens, model, latency
│   └── span: guardrail-check
│       └── attributes: passed=true, score=0.98
└── root attributes: conversation_id, user_id, total_cost, duration

Every span carries three things: a name, a kind (internal/LLM call/tool call), and attributes — key-value metadata. The two most important spans are the LLM call (what the model was asked, what it returned, at what cost) and the tool call (what the agent did to the world, and what came back). Parent-child relationships make the tree: the agent loop is the root, each model call and tool call is a child, and tool results are events or child spans of the model call that requested them. When you can replay this tree for any failed run, "debug the agent" becomes "read the trace."

OpenTelemetry GenAI Semantic Conventions

You do not need to invent a schema. The OpenTelemetry GenAI semantic conventions standardize exactly these spans so that any compliant tool — open-source backends, commercial observability platforms, or your own dashboard — can ingest your traces. The conventions you will use most:

Adopting the conventions costs nothing — they are attribute names — and pays off twice: your traces work with any GenAI-aware observability backend, and you avoid inventing a bespoke schema that every new tool refuses to understand. If you already emit standard OTel spans for your web services, agent spans are just more of the same pipeline.

Instrumenting the Agent Loop

Instrumentation is a wrapper around the two choke points every agent has: the model call and the tool call. A minimal instrumented loop:

from opentelemetry import trace
tracer = trace.get_tracer("agent")

def run_agent(messages, tools):
    with tracer.start_as_current_span("agent-loop") as root:
        root.set_attribute("conversation_id", ctx.conversation_id)
        for step in range(MAX_STEPS):
            with tracer.start_as_current_span("llm-call") as span:
                resp = llm.chat(messages, tools=tools)
                span.set_attributes({
                    "gen_ai.system": "dr-ai",
                    "gen_ai.request.model": "gpt-5-mini",
                    "gen_ai.usage.input_tokens": resp.usage.prompt_tokens,
                    "gen_ai.usage.output_tokens": resp.usage.completion_tokens,
                })
            if resp.tool_calls:
                for tc in resp.tool_calls:
                    with tracer.start_as_current_span("tool-call") as t:
                        t.set_attribute("gen_ai.tool.name", tc.name)
                        t.set_attribute("gen_ai.tool.arguments", tc.arguments)
                        t.set_attribute("tool.status", "ok")
                        result = execute_tool(tc)
                        messages.append(to_message(tc, result))
            else:
                return resp.content
        raise AgentStepLimit(MAX_STEPS)

Three instrumentation rules keep the traces useful rather than noisy. First, record prompts and outputs (as span attributes or events) — a trace without the actual prompt is a skeleton; you will re-fetch it from logs anyway, so record it once, respecting PII redaction. Second, capture tool errors as events, not just status attributes — the error message is usually the diagnosis. Third, add business context to the root span (conversation_id, user segment, feature flag) so you can search traces by what your users experienced, not just by trace ID.

Cost per Trace: Token Accounting as a First-Class Signal

Agent cost is the sum of its model calls, and it compounds with loops. Observability should answer, per trace and per user: how many model calls ran, how many tokens each consumed, what the total spend was, and — critically — which spans were the expensive ones. Implementation notes:

• Capture usage from every response (prompt_tokens, completion_tokens, cached tokens) and sum them up the span tree; store per-trace totals in root attributes.
• Apply your real unit prices when computing spend — store price per 1M tokens per model in a config, not in code constants, since providers change prices.
• Alert on runaway loops: step count, total tokens, and per-trace cost are the three alarms that catch budget incidents (a buggy tool that returns empty results can make an agent spin for 20 steps and $2 before you notice).
• Break cost down by component: model calls vs tools (tool latency is not billed but tool-retry storms inflate model calls).

Teams that track cost per trace also discover optimization opportunities organically: traces show the same system prompt being re-sent 12 times (enable prompt caching), or a 4k-token history dominating every call (summarize it). Cost telemetry is the byproduct of tracing done right, not a separate project.

Evaluation Integration: Traces as the Debug Interface

Agent quality is measured by evaluations — golden scenarios run against the agent with scored outcomes (task success, tool-use correctness, faithfulness, latency, cost). The integration between evals and tracing is the part most teams miss: every eval run should produce the same traces as production, linked by an eval_run_id, so a failing eval is not a score but a trace you can open.

# Eval harness: run scenario, score, and link to the trace
def eval_scenario(scenario, runner, judge):
    trace_id = start_eval_trace(scenario.id)      # same OTel pipeline
    result = runner.run(scenario.input)           # emits agent spans
    score = judge.score(scenario, result)         # LLM-as-judge or rules
    end_eval_trace(trace_id, {
        "eval.run_id": run_id,
        "eval.scenario": scenario.id,
        "eval.score": score,
        "eval.passed": score >= scenario.threshold,
    })
    return score

This pays off in the regression workflow: when a change drops the eval score, you diff the traces before and after — the tool call that now returns an error, the model that now refuses, the extra loop iteration that doubles cost. Trace-diffing is the agent equivalent of reading a stack trace, and it turns eval failures from a number into a diagnosis. Store eval traces with a retention policy (they are gold for future debugging) and let the same dashboards show production and eval traces side by side.

Debugging Workflows That Actually Work

With traces in place, debugging an agent becomes a repeatable workflow instead of an ordeal:

1. Reproduce and capture. Find the failing run by its trace (search by conversation_id or failure attribute) and open the span tree.
2. Walk the tree. Identify the step where behavior diverged: a model call with the wrong tool choice, a tool call with an unexpected error, a parsing failure after a tool result.
3. Inspect the boundary. 80% of agent bugs live at the model↔tool boundary: the model chose the right tool with malformed arguments; the tool returned valid JSON the model misinterpreted; the tool returned data the model treated as authoritative despite an error field. The trace shows both sides — arguments sent and result returned — which is exactly what you need.
4. Fix and re-run the eval scenario. The failing case belongs in your golden set as a regression test; the trace becomes the before/after evidence.

Two guardrails make the workflow safe: timeouts and step limits on every agent (a traced infinite loop is still an infinite loop), and trace sampling — record 100% of failed/expensive traces and a sample (e.g., 10%) of healthy ones; full-fidelity traces of every run are a storage and privacy liability. Apply PII redaction at instrumentation time: if you cannot ship a trace to your own dashboard, redact prompt content in the exporter, not after the fact.

One more habit separates teams that debug agents in minutes from teams that debug for days: searchable trace metadata. Attach every attribute you might filter on — user segment, feature flag, model, tool name, error code, eval score — at the root and child spans, and make trace IDs part of every user-facing error message and support ticket. When a user reports "the assistant told me my order was cancelled when it wasn't," support needs one search to land on the exact trace, and the engineer needs one click to see the tool payload the model misread. Cheap to add, invaluable in production.

Choosing Your Observability Stack

You have three realistic options, from fully managed to fully self-hosted:

The pragmatic stack for most teams: emit OTel GenAI spans from your agent code (the conventions work everywhere), ship them to a self-hosted or managed OTel backend for traces and cost, and add an eval harness that links to the same traces. That combination keeps you portable — you are never locked into a vendor's trace format, and switching dashboards is a config change. If you are on a budget, start with the open-source path; the instrumentation you write is identical to what a paid platform would require.

A pragmatic target for most teams: every production trace should answer three questions within ten seconds — what did the agent intend to do (the tool choice), what did the world return (the tool result), and what did it cost (tokens and dollars). If your trace view answers those three instantly, you have observability; if you are still grepping logs for one of them, keep instrumenting.

The Agent Observability Checklist

1. Emit a root span per agent run with business context (conversation_id, user_id)
2. Span per LLM call with gen_ai.* attributes: system, model, input/output tokens
3. Span per tool call with name, arguments, result, and error events
4. Use OpenTelemetry GenAI semantic conventions from day one
5. Sum tokens up the tree; store per-trace cost using real per-model prices
6. Alert on step count, token totals, and per-trace cost
7. Run evals through the same tracing pipeline; link eval runs to traces
8. Add timeouts and step limits to every loop
9. Sample traces (100% of failures, ~10% of healthy); redact PII at instrumentation
10. Diff before/after traces when eval scores regress

Agent observability is not a luxury layer on top of a finished agent — it is the scaffolding that makes agent development iterative at all. Trace the loop, watch the model↔tool boundary, account for every token, and your agent stops being a black box that occasionally misbehaves and becomes a system you can inspect, evaluate, and improve. For the API layer underneath, pair this guide with our AI API monitoring guide; for building the agents themselves, start with the agent development guide. And when you are ready to ship, DrAI's OpenAI-compatible gateway gives you usage telemetry across all your models in one place — sign in or see pricing.