Set it up
Recommended path: OpenTelemetry, with OpenInference's CrewAI and OpenAI instrumentations. You need an agent credential from Dashboard, Agents: one credential per agent.
1. Install the instrumentations and an OTLP exporter
The CrewAI instrumentation records the crew, its agents and its tools but no model calls, so add the one for the provider your crew calls. CrewAI 1.x calls OpenAI through the official openai package.
pip install openinference-instrumentation-crewai openinference-instrumentation-openai opentelemetry-sdk opentelemetry-exporter-otlp-proto-http2. Point the exporter at Toolcaise
export TOOLCAISE_AGENT_TOKEN=... # the agent's credential from Dashboard, Agents
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://toolcaise.com/api/v1/otlp/v1/traces
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$TOOLCAISE_AGENT_TOKEN"
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip
export OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none
# Keep prompts and outputs out of the export. Toolcaise never reads them either way.
export OPENINFERENCE_HIDE_INPUTS=true OPENINFERENCE_HIDE_OUTPUTS=true
# Optional, and unrelated to Toolcaise: turn off CrewAI's own anonymous telemetry.
export CREWAI_DISABLE_TELEMETRY=true3. Instrument before the crew starts
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
# Endpoint, credential and compression come from the OTEL_ environment variables above.
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
from openinference.instrumentation.crewai import CrewAIInstrumentor
from openinference.instrumentation.openai import OpenAIInstrumentor
CrewAIInstrumentor().instrument(tracer_provider=provider)
OpenAIInstrumentor().instrument(tracer_provider=provider) # model calls and token counts
# crew = Crew(agents=[...], tasks=[...])
# crew.kickoff()Then run the agent as usual. Its runs appear in the dashboard as soon as their first report arrives.
What Toolcaise receives
| How it arrives | |
|---|---|
| Runs | Each trace is one run. It is created when the first span arrives and completed, or failed, when the trace's root span ends. |
| Model calls | Model calls recorded by the OpenAI instrumentation become model spans with the model name and token counts. |
| Tool calls | Tool spans with the tool's name. |
| Steps | The crew kickoff, its agents and its tasks become steps. |
| Failures | A failed span in the trace adds an event that carries only the exception type, never its message. |
| Agent details | service.name, service.version and host.name describe the agent. |
What this path does not give you:
- Pause, cancel and approvals: OpenTelemetry only reports. Use the SDK's checkpoint() and guarded_action() for decisions.
- Cost: Toolcaise does not price tokens on this path, and stores cost only when the instrumentation reports llm.cost.total.
Everything here is reported by your agent or its framework and stored as agent-reported. Prompts, completions and tool arguments are never read: the OpenTelemetry endpoint reads only an allow-list of attributes, and the SDK sends metadata only.
Other ways
Other model providers
For Anthropic models add openinference-instrumentation-anthropic; for providers CrewAI reaches through LiteLLM, openinference-instrumentation-litellm. Instrument them the same way. Without one, the run shows the crew's structure but no model calls or token counts.
Approvals and pauses: the SDK around the crew
Wrap crew.kickoff() in a Toolcaise run and guard consequential tools with guarded_action, as in the LangChain quickstart. This reports a run of its own next to the trace's run.
Caveats
- Do not set OTEL_SDK_DISABLED to silence CrewAI's telemetry: it switches off every OpenTelemetry export in the process, this one included. Use CREWAI_DISABLE_TELEMETRY.
- CrewAI 1.15 requires openai 2.x; openinference-instrumentation-openai supports openai 2.8 and later.
- OpenInference captures prompts and outputs by default; the OPENINFERENCE_HIDE_ variables above stop that. We have not run this combination end to end.
What was checked
Checked on 2026-09-22 against crewai 1.15.22 (which calls the openai package for OpenAI models), openinference-instrumentation-crewai 1.1.18 and openinference-instrumentation-openai 0.1.60. The OpenAI instrumentation's output was part of Toolcaise's OTLP ingest tests; a crew was not run end to end.
Reference: SDKs, HTTP API, and the OpenTelemetry endpoint's limits and answers.