If everything is a "Critical" priority, then nothing is
There's a thread on r/sysadmin that's been rattling around my head for a couple of days. The title is the whole argument: "If everything is a 'Critical' priority, then nothing is." Three hundred-plus upvotes, ninety-odd comments, and every one of them is some version of "yep, lived it."
I've lived it too. Every place I've worked that drowned in pages had the same root cause, and it wasn't a tooling problem. It was that nobody was allowed to say a thing wasn't important.
How everything becomes critical
Nobody sits down and decides to label every alert P1. It happens by accretion.
A printer goes down once during a board meeting, someone gets yelled at, and now "printer offline" is a critical alert. A batch job fails silently and costs a day of reprocessing, so now every job failure pages the on-call. A customer complains that a dashboard was slow, so latency on an internal tool that six people use is suddenly the same severity as the payments API falling over.
Each individual decision is defensible. The sum of them is a pager that goes off forty times a shift, and an on-call engineer who has learned, correctly and rationally, that the fastest way to make the noise stop is to acknowledge and ignore. That's the trap. The desensitization isn't a character flaw. It's the only sane response to a system that cried wolf forty times yesterday.
The industry has a name for this, alert fatigue, and it's measurable. It shows up as slower responses, missed real incidents, and people quitting on-call rotations or quitting outright. xMatters and incident.io both rank burnout-driven attrition as one of the top costs of a noisy alerting setup, and it's the expensive one. You don't lose an evening, you lose the person who knew where the bodies were buried.
Severity has to mean something or it means nothing
The fix everyone reaches for first is "let's add more severity levels." That's backwards. The problem isn't that you have Critical and High and Medium. It's that the labels aren't tied to anything real, so they drift until they're decoration.
A severity level is a promise about response, not a description of how the author felt when they wrote the alert. Critical should mean: wake a human up, right now, because something a customer can feel is broken. If your definition of Critical can't survive the question "would I want to be woken at 3 a.m. for this?", it isn't Critical.
The most useful framing I've seen comes from the Google SRE workbook, and it's brutal in its simplicity. Every page should be actionable, and every page should require human intelligence. If a page only merits a robotic, scripted response (restart the thing, clear the queue), then automate that and don't page anyone. If a page is purely informational, it's not a page, it's a log line. By that filter, most "Critical" alerts in a fatigued system aren't critical. They aren't even alerts. They're anxiety with a delivery mechanism.
The same workbook puts a number on what sustainable looks like: roughly two actionable incidents per on-call shift. That's not a target to hit, it's a ceiling. You can only respond with genuine urgency a few times a day before the urgency burns off. If your on-call is fielding twenty pages a shift, you don't have an on-call rotation, you have a denial-of-service attack against your own team.
Alert on symptoms, not causes
The other half of the fix is what you alert on. Most noisy setups alert on causes: CPU is high, disk is at 80%, a specific host stopped responding. The trouble is that causes fire constantly and usually mean nothing. CPU spikes and recovers. One host in a healthy pool dies and the load balancer routes around it. You get paged, nothing was wrong.
Symptom-based alerting flips it. You alert on the thing the user experiences. Is the service returning errors? Is it slow past the point where the SLO says customers care? Are requests failing? Tie alerts to service-level objectives instead of individual metrics and the volume drops hard. One writeup I found while digging into this put the reduction at around 85%, while detection of real problems got better, not worse. Fewer pages, and the ones that survive matter. That's the whole game.
This is also the honest answer to the manager who insists everything stay Critical "to be safe." Symptom-based alerting is the safe option. A pager nobody trusts is the dangerous one, because the real outage arrives wearing the same costume as the four hundred false alarms before it, and your tired on-call waves it through.
The boring part nobody wants to do
None of this is a tooling purchase. You can do most of it in whatever you already have. The work is the boring, political part:
- Write down what each severity means in terms of response time and who gets woken. Put it in the runbook. Make it reviewable.
- Hold a monthly alert review. Pull the pages from the last 30 days. For each one ask: was this actionable? Did a human have to think? If the answer is no twice, that alert gets deleted or downgraded. Deleting alerts should be as normal as adding them. Most teams only ever add.
- Make "this wasn't a critical event" a safe thing to say. The cultural fix is the real fix. If nobody is allowed to downgrade the printer alert, you will be back here in six months.
A monitoring tool can help you hold the line once you've drawn it. A dead-simple heartbeat check that pages you only when a job that should have run didn't, instead of firing on every transient blip, is the kind of single-purpose, low-noise signal that earns its place on the pager. That's a chunk of why I built CronCanary the way I did: one alert, when the thing that matters silently stops, and nothing the rest of the time. But the tool is downstream of the decision. The decision is being willing to say "no, that's not critical," and then enforcing it.
The r/sysadmin thread is right. If everything is critical, nothing is. The fix isn't a better-sounding label. It's the discipline to let most things be ordinary, so that the rare genuine emergency still has the power to make someone move.