OpenTelemetry for AI agents: what to trace and how

Why tracing fits AI agents

An agent run is not one request. It is a loop: the model picks a step, the agent calls a tool, the result goes back to the model, and this repeats until the model answers or something gives up. Drawn out, that loop is a tree, with the run at the top and model calls and tool calls underneath it.

That tree is exactly what distributed tracing was built to record. OpenTelemetry gives you the tree in a vendor-neutral format, so the same instrumentation can feed whichever backend you choose. A log line can tell you a run was slow or expensive; a trace shows you which call made it so.

What a useful agent trace contains

Start with the structure and add detail only where it answers a question you will actually ask. For most agents that means:

• One root span per run, named after the task, so the trace’s start, end and status are the run’s start, end and outcome.
• One span per model call, with the model that answered, the input and output token counts, and the duration.
• One span per tool call, named after the tool, with its success or failure.
• Spans for meaningful agent steps, such as a planning phase, a graph node or a handoff to another agent, when your framework has them.
• Errors recorded on the span that failed, with the exception type, and each retry as its own span so you can count them.
• Cost, if your instrumentation prices the call. Otherwise record tokens and price them later against a price list you control.

run: triage-support-ticket              root span, OK
  chat example-model                    input 1,850 tokens, output 210
  execute_tool search_orders            OK
  chat example-model                    input 2,400 tokens, output 90
  execute_tool send_reply               ERROR TimeoutError
  execute_tool send_reply               OK (retry)

Resource attributes describe the process rather than the run: service.name for the agent, service.version for the release you deployed, host.name for where it ran. Set them once in the SDK and every span carries them.

OpenTelemetry GenAI semantic conventions and OpenInference

A trace is only useful across tools if everyone agrees what an attribute means. OpenTelemetry’s GenAI semantic conventions define those names. gen_ai.operation.name says what a span is (for example chat or execute_tool), gen_ai.request.model and gen_ai.response.model record the model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens record usage, and gen_ai.tool.name names the tool.

The conventions are still changing, and older instrumentation emits earlier names such as gen_ai.usage.prompt_tokens. A backend that reads agent traces has to accept both, so check which version your instrumentation follows before you build dashboards on the names.

OpenInference is a separate convention with its own family of instrumentation libraries for Python and JavaScript frameworks. It marks each span with openinference.span.kind (LLM, TOOL, CHAIN, AGENT and others), records the model as llm.model_name and records usage as llm.token_count.prompt and llm.token_count.completion. Both conventions ride on ordinary OpenTelemetry spans and OTLP, so choosing between them mostly comes down to which instrumentation exists for your framework.

What to keep out of agent spans

Many instrumentation libraries record the full prompt, the model’s output, and tool arguments and results as span attributes or events, and some do it by default. That is convenient while debugging and a liability everywhere else. Prompts carry customer messages, tool arguments carry account numbers, and a system prompt can contain an API key someone pasted in. Once text is in a trace, it is in every backend, bucket and export that trace reaches.

Treat content capture as something you turn on deliberately, for one agent and a short time, rather than as a default. Keep identifiers, counts, names and statuses, and drop text. OpenInference, for example, reads two environment variables:

export OPENINFERENCE_HIDE_INPUTS=true OPENINFERENCE_HIDE_OUTPUTS=true

Exception messages deserve the same suspicion. A failed HTTP call can put a whole request URL, query string included, into the message. The exception type is usually enough to group failures.

Sampling agent traces to control cost

A busy agent produces a lot of traces. Sample whole traces, decided at the root, so you never keep half a run. The standard sampler settings OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.25 keep roughly a quarter of runs and every span inside them.

Two cautions. Sampled traces undercount: if you sum token usage from a 25 percent sample, you see about a quarter of the spend, so keep every trace for agents whose cost you track, or record usage somewhere else. And head sampling drops failures and unusually long runs at the same rate as healthy ones. Tail sampling in an OpenTelemetry Collector decides after the trace ends, so it can keep every errored trace while sampling the rest.

Exporting agent traces over OTLP/HTTP

OTLP is OpenTelemetry’s wire protocol, and OTLP over HTTP is the most widely accepted way to send traces to a backend. The official SDKs read the same environment variables, so you can usually point an agent at a new backend without touching code:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is the full URL for traces. The general OTEL_EXPORTER_OTLP_ENDPOINT takes a base URL instead, and the SDK appends /v1/traces.
• OTEL_EXPORTER_OTLP_HEADERS holds comma-separated key=value pairs, URL-encoded, which is why a bearer token is written Authorization=Bearer%20 followed by the token.
• OTEL_EXPORTER_OTLP_PROTOCOL selects http/protobuf, http/json or grpc. The default differs between SDKs.
• OTEL_EXPORTER_OTLP_COMPRESSION set to gzip is worth it for batches of agent spans.
• OTEL_SERVICE_NAME sets service.name on everything the process exports.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Endpoint, credential and compression come from the OTEL_ environment variables above.
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

Frameworks that already emit agent traces

You rarely need to write these spans by hand. Instrumentation exists for the common agent frameworks and produces the tree described above:

• LangChain and LangGraph: OpenInference’s LangChain instrumentation hooks langchain-core, so chains, agents and graph nodes become spans, with model and tool spans inside them.
• CrewAI: OpenInference’s CrewAI instrumentation records the crew, its agents, tasks and tools, but not model calls. Add the instrumentation for the provider the crew calls, such as OpenAI, Anthropic or LiteLLM, to get model spans and token counts.
• OpenAI Agents SDK: openinference-instrumentation-openai-agents turns the SDK’s own traces into OpenTelemetry spans for agents, model calls, tools and handoffs.
• Vercel AI SDK: telemetry is built in. In AI SDK 7 you register it once with @ai-sdk/otel, and each generation step and tool call becomes a span.

Check each one’s content settings before you ship. OpenInference captures prompts and outputs by default, and the AI SDK records inputs and outputs unless each call sets recordInputs: false and recordOutputs: false.

The gap between tracing and control

A trace is a record of what already happened. It can show that a run looped through dozens of model calls, but by the time the span is exported those calls have been paid for. Nothing in OpenTelemetry can pause a run, refuse a tool call or stop a process: the data only flows out of the agent.

Control has to sit where the agent’s actions pass through before they happen. That can be a check inside the agent’s code between steps, a gate in front of consequential tools that waits for a person, or a proxy between the agent and the model provider that can refuse the next call. The trace tells you where you need limits and approvals; something else has to enforce them.

Sending agent traces to Toolcaise

Toolcaise accepts OpenTelemetry traces over OTLP/HTTP, so an agent that is already instrumented needs no Toolcaise package. Create a credential for the agent on the Agents page of the dashboard, then set three exporter variables and switch off metrics and logs export, which Toolcaise does not accept:

read -rsp 'Agent credential: ' TOOLCAISE_AGENT_TOKEN; echo
export TOOLCAISE_AGENT_TOKEN
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://toolcaise.com/api/v1/otlp/v1/traces
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$TOOLCAISE_AGENT_TOKEN"
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip
export OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none
# Then start your agent as you normally do.

The endpoint accepts protobuf or JSON bodies, compressed with gzip or not. For the Vercel AI SDK, the same URL and an Authorization: Bearer header go to its trace exporter in code; the Toolcaise quickstarts have the setup for each framework above. Once spans arrive, Toolcaise maps them like this:

• Each trace becomes one run. It is created when the first span arrives and marked completed or failed when the trace’s root span ends. Once finished, a late or retried batch cannot change it.
• Model spans keep the model name and the input and output token counts, from either the GenAI or the OpenInference attributes.
• Tool spans keep the tool’s name. Chain, agent and graph-node spans become steps, so a LangGraph run shows its nodes in order.
• A failed span adds an event that carries only the exception type, never its message.
• service.name, service.version and host.name describe the agent.

Privacy is enforced when the request is decoded. Toolcaise reads a fixed list of attributes, such as model names, token counts and tool names, and skips every other attribute without decoding it, so prompts, completions, tool arguments and exception messages are never read even if your instrumentation sends them. A trace with no model, tool or agent spans in it, such as an unrelated HTTP request sent through the same exporter, is ignored.

What the OpenTelemetry path does not cover

This path only observes. Toolcaise cannot pause, stop or ask for approval on a run it learns about through traces, because the spans arrive after the work is done. For that, use the Toolcaise SDK’s checkpoint() between steps and guarded_action() around consequential tools, or run the agent under Toolcaise Connect. Connect is a local CLI whose gateway measures model calls itself and stops the command at a runtime, token, cost or model-call limit, and with --remote-control it lets the dashboard pause or stop the run. A run reported through the SDK appears next to the trace’s run; Toolcaise does not merge the two.

• Cost: Toolcaise does not price tokens on this path. It stores a cost only when the instrumentation reports llm.cost.total.
• Trust: token counts are what your instrumentation reported. They are the agent’s claim, not an independent measurement.
• Traces only: the metrics and logs endpoints deliberately answer 501, so discarded data never looks like success.
• Distributed traces: a trace whose root span belongs to another service never completes in Toolcaise, and its run is flagged as stalled after 30 minutes without updates.