How to detect silent failures in scheduled AI agents

Why scheduled agents fail without an error

An agent that crashes loudly is the easy case. The harder one is an agent that simply stops producing work: nothing errors, because nothing ran, and a monitor that only watches for errors sees what it sees on a normal day.

Scheduled agents tend to go quiet in four ways:

• The scheduler stopped. A cron entry was lost when a machine was rebuilt, a timer was disabled, or someone switched off a CI schedule. The agent’s code is fine; it is never started.
• The host is gone. The machine was shut down, filled its disk or lost its network, so there is no process left to report anything.
• The run started and never finished. It is waiting on a network call with no timeout, or it was killed for using too much memory before it could report an end.
• The run succeeded and produced nothing. The process exited cleanly, but the file is empty, the upstream query returned no rows, or the result went somewhere nobody reads.

Each needs its own check, and alerting on errors catches none of them.

Monitor the expected schedule with a dead man’s switch

The answer to work that never starts is to alert on absence. A dead man’s switch expects a signal by a deadline and raises an alarm when the deadline passes without one. For a scheduled agent, the signal is the start of a run and the deadline is the scheduled time plus a grace period.

Two details decide whether it works. First, the check has to live outside the thing being watched, because a dead scheduler cannot report that it missed a run. Second, the monitor needs the next expected time, not just the cron expression, so each run should report when the following one is due. If those reports stop, the last deadline stays in place and still fires.

Choose the grace period from how the job really behaves. A job that starts on time can use a short window; one that waits for an upstream export needs a longer one, or it will alert every morning.

Use heartbeats to tell whether the host is alive

A heartbeat is a regular check-in from a process or a machine. When it stops, you learn the host or process is gone, which a schedule check only discovers at the next slot. A heartbeat does not show that work happened, though. A process can check in every minute while its task loop is stuck. Use heartbeats for liveness and the schedule check for coverage.

Catch runs that start and never finish

A run that reports a start and then nothing else is easy to miss, because it looks like it is running. Treat a run as stalled when neither the run nor any of its steps has been updated for longer than a normal run could take. Reporting individual steps makes this reliable: a run working slowly through fifty documents keeps updating, while a stuck one does not.

Set timeouts inside the agent as well, on every network call and on the run as a whole. A stall alert tells you something is stuck; a timeout turns it into a failure with a reason attached.

Treat repeated failures as their own signal

One failed run of a nightly agent may be a flaky upstream API. Three in a row usually means something changed: an expired credential, a renamed field in a source system, or a model version that was retired. Alerting on consecutive failures rather than on every failure keeps single transient errors out of the channel while still catching a job that has stopped working.

Check the output, not only the exit status

A clean exit only tells you the process ended. Look at what it produced; a few cheap checks catch most problems:

1. The output exists where the next step reads it.
2. It is not empty.
3. It is fresh: written within the expected window, not left over from yesterday.
4. It has the expected shape, for example a JSON object with a summary field.

None of these says the content is right. For that, have a person review outputs, or a sample of them, and record the decision separately from the run’s status.

Route alerts to someone who acknowledges them

An alert needs a named recipient and a place they will see it: a team channel for problems that can wait until morning, a paging tool for ones that cannot. Record acknowledgement so the team can see someone is looking, and keep it separate from the condition clearing. An acknowledged incident may still be broken, and a recovered one may still deserve a look at why it broke.

Escalate what nobody has acknowledged: after an hour, resend it at a higher severity or to a paging tool rather than wait for someone to scroll back.

Avoid alerts for agents that are idle between runs

Scheduled agents are idle most of the time, and a naive liveness check treats idle as dead. If an agent only sends heartbeats while it runs, a ten-minute heartbeat check fires after every run and clears at the next one. After a week of that the channel is muted, and the real alert is missed.

Some ways to keep the signal clean:

• Send liveness from something that is always running, such as the host or a supervising process, rather than from the short-lived job.
• Rely on the schedule check for coverage, and use heartbeats for processes that are meant to stay up.
• Silence a destination during planned maintenance instead of switching the checks off.
• Send recoveries and digests to a channel, and page only for problems nobody has acknowledged.

How Toolcaise detects silent failures

Toolcaise opens an incident for four conditions. The server checks each workspace about once a minute, and again when someone opens the dashboard.

• Stale heartbeat: no heartbeat from the agent for 10 minutes, or for the check-in interval set on the agent’s page. Any report from an agent counts as a heartbeat, and a run reported through the SDKs sends one every 60 seconds.
• Missed schedule: an enabled schedule whose next expected run is 15 minutes overdue and has not been triggered.
• Stalled run: a run still marked as running with no update to the run or any of its steps for 30 minutes.
• Repeated failures: the agent’s three most recent finished runs all failed.

An incident moves from open to acknowledged to closed as people act on it, and recovers by itself when the condition clears. If the condition returns, the same incident reopens and its occurrence count goes up, so a flapping job shows as one incident with a history. The missed-schedule check depends on the agent reporting its schedule, including the next expected run, which is simplest to send at the start of each run:

from toolcaise import Toolcaise

client = Toolcaise()
client.report(schedule={
    "name": "nightly-research",
    "calendar": "0 6 * * *",
    "timezone": "America/New_York",
    "enabled": True,
    "active": True,
    "lastTriggeredAt": "2026-09-23T10:00:00Z",  # when this run started
    "nextRunAt": "2026-09-24T10:00:00Z",        # when the next run is due
})

Alerts go to the destinations a workspace sets up: email, Slack, Microsoft Teams, Discord, PagerDuty or an HTTPS webhook signed with HMAC-SHA256. Each destination has a test send, a minimum severity and an optional maintenance window of up to 30 days. An incident is sent as a warning when it opens, as critical if nobody has acknowledged it within an hour, and again when it is acknowledged, recovers or is closed. PagerDuty receives these as trigger, acknowledge and resolve events under one key per occurrence, so it keeps a single incident. An alert still waiting to be sent is cancelled if the condition clears first.

Toolcaise treats two kinds of agent as idle, not offline, between runs. An agent that has reported a schedule is expected to be quiet until its next run, so it gets no stale-heartbeat incident while that run is not yet overdue; a run that fails to start is the missed-schedule incident instead. Agents that Toolcaise Connect runs or watches, with toolcaise-connect run, behind its MCP gateway, through its Claude Code and Codex adapters or as a program it found, are idle while the machine’s Connect service keeps checking in, and are flagged only if the machine has also been silent for 10 minutes.

For output, the SDKs’ output_checks() can check that a file on the agent’s host exists, is not empty, is fresh and matches a small JSON Schema subset, and reports only the results, never the path or contents. With Connect, toolcaise-connect run --output PATH checks that the file exists and is not empty after the command exits. A failed check opens a warning alert that goes to the same destinations.

What Toolcaise does not detect

Toolcaise sees only what is reported to it. If an agent never reports a schedule, a run that never starts is not flagged as overdue, and Connect does not report a schedule for a wrapped command, so send one through the API or an SDK. Apart from the heartbeat, the thresholds are the same for every agent, so a run whose single step takes longer than 30 minutes, with nothing else reported in that time, will be flagged as stalled.

An agent that exits between runs and has reported neither a schedule nor a Connect wrapper looks the same as a dead one once its heartbeats stop, so it gets a stale-heartbeat incident 10 minutes later. Report its schedule, set how often it should check in on its page, or run it under Connect, to avoid that. Output checks confirm a file’s existence, size, age and shape, not that its content is correct. A delivered alert means the provider accepted it, not that anyone read it.