See every run, step and cost

What you see for every agent

Each agent in the workspace has a health status, its last heartbeat, 24 hours of activity and the credential it reports with. Open it and you get its runs, newest first.

• Runs with a status (running, completed, failed, held or canceled), when they started and how long they took.
• Steps inside a run, reported as spans: a model call, a tool call or a named step, each with its timing and whether it succeeded.
• Model and token counts per call, and cost where your prices are known.
• Events the agent reports along the way, such as a limit being reached or a tool call waiting for approval.
• Outcomes the run reports at the end, which a person can then accept or reject.

Everything an agent sends is labelled as agent-reported. Model calls made through Toolcaise Connect’s gateway are different: Connect counts them itself from the provider’s usage figures instead of taking the agent’s word for them.

Connect an agent in a few lines

Most agents are connected with an SDK. The client records the run, its spans and the model calls made through an instrumented OpenAI client. It never sends prompts or model output.

from openai import OpenAI
from toolcaise import Toolcaise

client = Toolcaise()  # reads TOOLCAISE_AGENT_TOKEN
openai = client.instrument_openai(OpenAI())

with client.run("nightly-research"):
    with client.span("collect sources", kind="tool", tool="web_search"):
        sources = collect_sources()
    openai.chat.completions.create(model="gpt-4.1", messages=[...])
    client.outcome("report written", status="succeeded")

If your framework already emits OpenTelemetry traces, point its exporter at Toolcaise instead, with no Toolcaise package. If the agent is a command on a machine you control, wrap it with Toolcaise Connect and you also get enforced limits.

Why step-level history matters

When a run fails, the question is almost never whether it failed. It is which step failed, on what input, after how many retries, and what it had already done. A run with named steps answers that in one screen. A run that only reports start and finish sends you back to raw logs, and a heartbeat alone tells you the process was alive, not that the work happened.

Usage and cost beside the work

Token counts come from the provider’s usage figures on each call. Cost is calculated from prices you supply, so a model without a price shows tokens but no cost rather than a guess.

Reporting never takes your agent down

The SDKs queue reports in memory and send them from a background thread. If Toolcaise is slow or unreachable, your agent carries on and the reports follow, retried with backoff. Only if the queue fills (1,000 reports by default) are reports dropped, with a warning in your agent’s logs.