Agent monitoring and agent supervision answer different questions

Four questions to ask about any agent

Anyone who runs an AI agent in production ends up asking four questions about it, usually in this order:

• Is it alive? The process is running, reachable and not stuck.
• Did it do the work? The run that was expected actually started, finished and produced something.
• Was the work good? The output is accurate, complete and fit for the person who will use it.
• May it take this action? A specific, consequential step is allowed to happen now.

They can sound like one question asked four ways, but each needs different evidence, and the last two usually need a person. Monitoring is built to answer the first question well and part of the second. Supervision is the practice of answering all four, with a named owner for each decision.

What agent monitoring covers

Monitoring collects what the agent and its runtime report: heartbeats, process health, run starts and ends, step timings, model calls, token counts, latency and errors. Good tracing, often called agent observability, lets you reconstruct a run after the fact and find the step that failed. That is valuable, and everything else in this guide builds on it.

Its blind spot is expectation. Monitoring records what happened and has no opinion about what should have happened. A run that never started produces no trace at all, and a run that finished with a useless result looks the same as one that finished with a good one. A monitoring view also has no way to say no before an action runs.

What agent supervision adds

Supervision starts from the monitoring record and adds the parts that need a decision:

• Expectations: a declared schedule and a definition of the output, so missing work becomes a visible condition instead of an absence nobody notices.
• Judgment: a person reviews what the agent produced and records whether it was acceptable, separately from the agent’s own claim of success.
• Authority: specific actions wait for a decision before they run, and someone can stop or limit a run that is going wrong.

This is the practical difference between observability and oversight. Observability means you can explain an agent’s behavior. Oversight means someone is accountable for what it does. Detailed traces make oversight cheaper, because the reviewer has context, but they don’t perform it.

Match each question to its evidence and owner

A useful way to design supervision is to write down, for each question, what evidence answers it and who acts on the answer. For a typical scheduled agent it might look like this (an example, so adjust it to your workflow):

• Is it alive: a recent heartbeat or process check. Acted on by whoever runs the host.
• Did it do the work: the expected schedule compared with the runs that actually started and ended, plus a reported outcome that points at the artifact. Acted on by the workflow owner.
• Was the work good: the artifact itself, read against the brief, with the reviewer’s decision and reason recorded. Acted on by a named reviewer.
• May it take this action: a request that describes the exact operation, decided before it runs. Acted on by the person accountable for the consequence.

The evidence moves further from the agent as you go down the list. A heartbeat comes from the agent. A quality decision can’t, because the agent is the party being checked.

Why one green status hides decisions

Many dashboards roll everything into a single health indicator. That works until the questions disagree, which happens often. Here are three examples of a green agent with a real problem:

• The heartbeat is fresh, but Monday’s scheduled run never started because a cron entry was removed during a deploy.
• The run completed without errors, but the report cites last month’s data because a source quietly moved.
• Every step succeeded, including the one that emailed a draft to a customer list instead of the internal reviewer.

The reverse happens too: a heartbeat goes stale during a network blip while the work is delivered on time. When one status stands in for four questions, people learn either to ignore it or to trust it too much. Keeping the answers separate lets each one be wrong without hiding the others.

How teams grow from monitoring into supervision

Nobody needs all of this on the first day. Most teams add layers in roughly this order as an agent takes on work that matters more:

1. Instrument runs and steps, so a failure can be traced to the step that caused it.
2. Declare the schedule and alert on missing work, so silence becomes an incident rather than something discovered days later.
3. Report outcomes and review them, starting with the workflows whose output other people rely on.
4. Put approval in front of the few actions that are irreversible, external, costly or sensitive.
5. Enforce limits from outside the agent, such as a cost ceiling or a runtime limit that ends the process, so a runaway run doesn’t depend on the agent’s cooperation to stop.

Each layer pays off once the one before it is in place. Approvals on an agent whose runs you can’t see are hard to judge, and outcome reviews without a declared schedule miss the runs that never happened.

A worked example with one recurring agent

Here is one agent taken through all four questions. It is an illustrative example, not a customer case. A compliance team runs an agent every Monday at 06:00. It collects the previous week’s regulatory filings, writes a two-page brief and emails it to a 40-person distribution list.

• Alive: the agent sends heartbeats while it runs. If the host dies mid-run, the heartbeat goes stale and whoever runs the host is alerted.
• Did the work: the agent reports its schedule, including the next expected run. If nothing has started by 06:15 on a Monday, that is an overdue schedule for the workflow owner, even though nothing has technically failed.
• Was it good: the run reports an outcome with a digest of the brief. A compliance analyst opens the brief, checks a few filings against their sources, and accepts it or rejects it with a reason such as ‘missed two filings from the state regulator’.
• May it act: sending to the distribution list is guarded. After the review, the analyst approves that exact send (this brief, this list), and the approval can be used once.

Written out like this, the gaps are easy to see. If nobody is named as the reviewer, the third question has no owner. If the send isn’t guarded, a bad brief reaches 40 people before anyone reads it.

Where Toolcaise fits

Toolcaise is organized around these four questions. Agents report through the Python or Node.js SDK, plain HTTP, OpenTelemetry traces or Toolcaise Connect, and runs, steps, token usage and outcomes are kept together for each agent.

For missing work, Toolcaise opens an incident for a stale heartbeat (10 minutes without a report, or the check-in set for the agent), an overdue schedule (15 minutes late), a stalled run (30 minutes without an update) or three consecutive failed attempts, and can notify a Slack channel, a signed HTTPS webhook or email. For quality, agents report an outcome with a status and an optional artifact digest, and workspace owners and admins accept or reject it. Every review is kept, separately from what the agent claimed.

For actions, the SDKs’ guarded_action asks a person to approve the exact action and uses that approval once. For limits that don’t rely on the agent’s cooperation, toolcaise-connect run measures model calls on your machine and stops the command and everything it started at a runtime, cost, token or call limit. With --remote-control, the dashboard can pause, resume or stop that run.

What Toolcaise does not cover

Toolcaise does not judge whether an agent’s work is correct. That is the reviewer’s decision, and Toolcaise records it. Everything an agent reports is stored and shown as agent-reported, so it is the agent’s claim rather than independent proof that an artifact exists or was delivered.

Visibility depends on the integration: a tool call your code never reports won’t appear. Pause, cancel and approvals through the SDKs are cooperative, so they work at the checkpoints your code calls and do nothing for code that skips them. Incident checks run about once a minute per workspace, which is prompt but not instantaneous.