Dead Man’s Switch

Pattern: A named solution to a recurring problem.

Make a scheduled job check in with an independent monitor, then alert when the expected proof of completion doesn’t arrive on time.

Also known as: Heartbeat Monitoring, Missing-Run Detection

The name comes from a safety control that requires a machine operator to keep proving they are present. Release the control, and the machine stops. Software reverses the polarity: a scheduled job must keep proving that it ran. Silence isn’t treated as health.

Understand This First

• Silent Failure — the failure class this pattern makes visible.

• Observability — the monitoring substrate that receives and routes the signal.

Context

This is an operational pattern for unattended work. It covers backups, billing exports, cleanup tasks, dependency scans, nightly evals, and scheduled coding agents. The scheduler may accept a job even when the machine, credentials, network, or process fails before useful work finishes.

Ordinary error reporting covers the path where code runs far enough to detect its own failure. It can’t report that the scheduler never launched the job. It may also miss a process that hangs before reaching its error handler. Those gaps matter more when teams leave agents running on timers and expect useful artifacts hours later.

Problem

A scheduled job can disappear without producing an error where anyone looks. The host may be down. A timer may be misconfigured. An expired credential may prevent startup. A job may begin, wedge on a tool call, and never reach its final log line. In each case, the missing output is the evidence, but most monitoring systems wait for an emitted failure.

How do you detect a run that never starts, never finishes, or vanishes before it can report an error?

Forces

• Silence is ambiguous. No signal can mean success, delay, failed delivery, or a job that never ran.

• Schedules have normal variation. Queueing, rate limits, and variable workloads make an exact deadline too brittle.

• The monitor must survive the monitored system. A heartbeat stored only on the same host disappears with the host.

• Completion isn’t correctness. A success check-in proves the job reached a point in its control flow, not that its output was useful.

• Alerts can train people to ignore them. A grace period that’s too short turns harmless delay into noise.

Solution

Have the job send a success check-in to an independent monitor, and give the monitor a deadline plus a grace period. If the check-in doesn’t arrive, the monitor alerts the owner or creates bounded recovery work. The monitor watches for expected presence, so a missing execution becomes a visible state rather than an absence of evidence.

Put the success check-in after the job’s required work and verification. A ping sent at launch proves only that the scheduler started something. For jobs that can hang or run for a long time, send a distinct start signal as well. The monitor can then distinguish three cases: no start, started but overdue, and completed successfully.

Choose the deadline from the real schedule and expected run time. Add enough grace for ordinary queueing and runtime variation, but not enough to hide a meaningful miss. If an hourly job usually finishes in eight minutes, a 20-minute grace period leaves room for normal variation. A five-second grace period would page on noise; a six-hour grace period would defeat the point.

Keep the check-in path outside the system it watches. A local timestamp file can’t tell you that the machine is offline. An independent service, host, or control plane can. Keep the ping quick and bounded. Retry it only a few times so a brief network fault doesn’t create a false alarm or block the job.

Route the alert somewhere with an owner and a response. A runbook should say how to check the scheduler, host, credentials, last known run, and partial artifacts. High-confidence, low-risk failures may feed Autonomous Remediation, but the switch itself should only detect and report. It shouldn’t invent a recovery policy.

Warning: A success check-in is a liveness claim, not a quality judgment. Keep separate checks for semantic correctness: validate the backup, inspect the report, run the eval, or verify the agent’s patch. A job can ping successfully and still produce garbage.

How It Plays Out

A repository runs a maintenance agent every night to update dependencies. The agent opens a pull request only when the changes are safe. Its wrapper sends a start signal before launch and a success signal after tests pass and the pull request exists. One night the agent wedges while waiting for a package registry. The monitor sees the start without a timely success, marks the run overdue, and opens an incident with the run identifier and last log location. The team learns about the missing change before the next day’s release window.

A data platform exports a customer billing file at 02:00. The export script used to email on exceptions, but a broken timer produced no exception because the script never started. The team adds an external check due at 02:30 with a 15-minute grace period. When a later configuration change disables the timer, the missing check-in pages the owner at 02:45. The alert’s runbook confirms that no partial file exists before restarting the task.

A weekly cleanup job often finishes between 40 and 70 minutes because its input size varies. The first heartbeat configuration allows only 45 minutes, so harmless long runs trigger repeated alerts. The team measures the distribution, sets a realistic maximum runtime, and keeps a separate alert for abnormal growth in duration. The switch now detects absence without pretending that every slow run is a failed run.

Consequences

Benefits. The pattern detects failures that error logs can’t: missed starts, dead hosts, broken schedulers, expired startup credentials, and hung runs. It gives unattended agent work a clear proof-of-life contract. Start and success signals separate scheduling failures from execution failures, which shortens diagnosis. An independent monitor can also report the loss of the entire execution environment.

Liabilities. Every monitored job gains another configuration surface: schedule, time zone, deadline, grace period, check-in endpoint, notification route, and owner. Bad timing assumptions create false alarms. A failed network request can make a healthy job look dead, while a misplaced success ping can make a broken job look healthy. Check-in URLs may act as credentials and need secret handling. The pattern also adds a new service whose own availability matters, so teams need a policy for delayed or missing alerts from the monitor itself.

The deepest limit is semantic. A dead man’s switch can prove that a control-flow point was reached. It can’t prove that an agent made the right change, that a report contains correct numbers, or that a backup can be restored. Pair liveness monitoring with Fail Fast and Loud inside the job and with output verification after it.

Detects: Silent Failure — A missing check-in exposes the silent failure that occurs when a scheduled job never starts or never finishes.

Enables: Autonomous Remediation — A trustworthy missed-run signal can start bounded remediation when the recovery path is known and safe.

Enables: Runbook — A missed deadline can trigger the runbook for diagnosing the scheduler, host, credentials, or wedged job.

Extends: Fail Fast and Loud — Fail Fast and Loud reports errors from work that ran; a dead man's switch also catches work that emitted nothing.

Specializes: Observability — The switch applies observability to the expected presence of a scheduled execution rather than to an emitted error.

Used by: AgentOps — AgentOps can monitor check-in deadlines across a fleet of scheduled agents and route missed runs to responders.

Used by: Background Agent — Unattended background agents use check-ins to prove that their scheduled work started and completed.

Sources

• The Healthchecks.io cron monitoring documentation describes heartbeat monitoring as a dead man’s switch and documents completion pings, grace periods, start signals, explicit failure signals, and abnormally long runs.

• Tuft & Needle’s DeadMan implementation is an early open-source example of monitoring recurring jobs by alerting when their expected check-ins stop arriving.