Your scheduled jobs can fail silently — and nobody gets paged

2026-06-07 · ~6 min read · Conor Dobbs

Between June 1 and June 4, GitHub's Dependabot stopped creating scheduled version-update pull requests on time. Delays grew over the window and topped out at two days. Around 1.5 million repositories with active scheduled-update configs were affected. The root cause was a change to an internal service that routes requests for Dependabot and a few other systems.

Here is the part worth sitting with: nothing crashed. No 500 page. No red status banner for most of it. The scheduled jobs were not erroring — they were quietly running late, and a late job throws no exception. If GitHub can let a scheduled pipeline stall for two days across a million-plus repos, your cron jobs can do the same thing on a Tuesday and you will find out from a customer.

Why a missed run is the hardest failure to see

Most monitoring watches for the wrong shape of problem. You alert on error rates, on 5xx responses, on latency spikes, on a queue backing up. Each of those fires when something runs and goes wrong.

A scheduled job that never starts produces none of those signals. There is no request, so there is no error. There is no log line, because the code that writes the log line never executed. Your dashboards stay green because green is the default state of a thing that did not happen. The failure is an absence, and you cannot alert on an absence by watching for presence.

This is exactly the class of failure that bites the jobs you depend on most:

Nightly backups. The backup cron dies. You notice the day you need to restore.
Billing and invoicing runs. The monthly charge job skips a cycle. Revenue goes missing and no error is logged.
Data syncs and ETL. The sync that feeds your reporting warehouse stops. Numbers look plausible because they are yesterday's.
Certificate and token renewals. The renew job fails quietly. Two weeks later TLS expires in production.
Cleanup and retention jobs. Disk fills, or stale data piles up, until something downstream breaks loudly.

In each case the gap between "it stopped" and "I found out" is measured in days, and the cost of the gap grows the whole time.

The pattern that catches it: a dead-man's-switch

You flip the logic around. Instead of waiting for the job to report a failure, you require the job to report a success, and you alert when that report does not arrive.

The mechanics are small. At the end of a successful run, the job pings a URL:

# end of your nightly backup script
pg_dump mydb | gzip > /backups/db-$(date +%F).sql.gz \
  && curl -fsS https://your-monitor/ping/nightly-backup

An external watcher knows the job is supposed to check in once a day, give or take a grace window. When a check-in is late, the watcher pages you. The job does not have to detect its own failure — its silence is the alert. That is the whole trick, and it is the pattern that turns a non-event into a signal.

Three properties make this work where ordinary monitoring does not:

It is external. The watcher runs nowhere near your job. If your whole box is down, your VPC is unreachable, or the cron daemon itself died, an in-process health check goes down with the ship. A separate service notices the missing ping regardless.

It measures the schedule, not the code. You are not asserting the backup is good. You are asserting it ran on time. That is a weaker claim, and a much more reliable one to enforce, and it catches the entire "never started" failure mode.

It has a grace window. Real jobs run a little long sometimes. A good monitor lets you say "expected hourly, page me only after 90 minutes" so you get caught failures without getting paged for noise.

What I learned wiring this up

A few things that are not obvious until you have run it for a while:

Ping on success, not on start. If you ping when the job begins, a run that starts and then dies halfway looks healthy. Put the ping on the last line, after the work is done and verified.

Separate the steps that matter. A backup-and-upload job has two failure points. Ping after the upload confirms, not after the dump. If you only check the dump, you will trust a backup that never left the box.

Tune the grace window to reality, not hope. Watch the actual run durations for a week, then set the window above the worst case you saw. Too tight and you train yourself to ignore the alerts; too loose and a real outage burns hours before it pages.

Alert routing is the point. A monitor that emails an inbox nobody reads is theater. Route the page to where you will respond — SMS, a phone call, the on-call channel.

The takeaway

The GitHub incident is a useful reminder precisely because GitHub is good at this. They have world-class observability and it still took meaningful time to surface a two-day scheduling stall, because a delayed job is genuinely hard to see from the inside. Your setup is not better than theirs. The honest move is to assume your scheduled jobs will fail silently at some point, and to put a watcher in front of the ones whose silence would cost you something.

Pick your three most important recurring jobs — the backup, the billing run, the sync that feeds your numbers. Add a one-line ping to each. Point it at an external monitor with a grace window. That is an afternoon of work that converts your worst class of failure from "a customer tells me" into "I get paged in minutes."