There is a moment in every on-call rotation where an engineer stops reading the full alert body. They glance at the title, see a pattern they recognize as almost certainly noise, and acknowledge it without investigation. This takes about four seconds. No incident ticket, no follow-up, no corrective action. It just gets silenced.
That moment feels harmless. It is the engineering equivalent of skimming past junk mail. But stack enough of those moments across a team over a quarter, and you have built something more serious: a culture where alerts are treated as background radiation instead of meaningful signals.
The direct cost is measurable, but rarely measured
Take a straightforward calculation. A mid-level SRE in a mid-size platform team costs roughly $85-100 per working hour in total compensation and overhead. A typical noisy monitoring environment generates somewhere in the range of 8-15 actionable-looking alerts per engineer per week that turn out to be false positives. Each one takes between 4 and 20 minutes to triage, depending on whether it requires cross-referencing runbooks or checking deployment history.
That is 30-90 minutes per engineer per week spent on alerts that should never have fired. For a team of six SREs, you are looking at 3-9 engineer-hours per week consumed by noise. That comes to roughly 150-450 engineer-hours per year, or the equivalent of 1-3 months of a full-time hire, doing nothing but dismissing bad alerts.
We are not saying this math proves a particular ROI for any given tool. What it does show is that noise has a budget, and that budget is currently invisible to most engineering managers because it never appears as a line item anywhere.
The trust erosion problem is harder to quantify but more damaging
The dollar figure above is real, but it is not the deepest problem. The deeper problem is what alert fatigue does to the team's relationship with its monitoring infrastructure.
When engineers start treating alerts as probably-noise-unless-proven-otherwise, their response latency to real incidents increases. Not dramatically at first. They still triage. But the cognitive posture has shifted: the default assumption is "this will turn out to be nothing." That bias introduces a delay, even if it is only 90 seconds of hesitation before taking the alert seriously.
In a service with tight SLO burn rate requirements, 90 seconds of delay on a real P1 can be the difference between a 10-minute incident and a 45-minute incident. That gap doesn't show up in postmortem root cause analysis, because the delay was not an action, it was an inaction born from a prior pattern of disappointment.
Consider a pattern we saw with a fintech platform team last year. Their Kubernetes cluster was generating CPU throttling alerts at a rate of about 40 per day across various services. The team had long since determined that roughly 38 of those per day were self-resolving within 2 minutes and represented no user-facing impact. They stopped logging tickets for them. Two weeks later, a real CPU resource contention issue caused a 12-minute latency degradation in their payment processing API. The alert fired. It looked identical to the noise they had been silencing for months. The engineer on-call acknowledged it, made a note to investigate later, and the degradation continued for another 22 minutes before someone noticed it in a customer escalation.
On-call rotation health is downstream of alert quality
There is another cost that is almost never attributed to alert fatigue in postmortems: rotation churn. Engineers leave on-call rotations, and sometimes leave teams entirely, because sustained noisy on-call is grinding work that produces no learning and no satisfaction.
The on-call experience should, in the best case, be a concentrated way to understand how a system behaves under stress. When most pages are noise, that learning signal disappears. What remains is the stress without the reward of understanding. Engineers who care about learning will find that unsatisfying. Over 12-18 months, that translates into turnover in your most operationally experienced team members.
Replacing an SRE with deep system knowledge costs more than the obvious recruiting and ramp time. You also lose the institutional knowledge of which alerts are trustworthy, which thresholds are calibrated well, and which runbooks actually reflect current system behavior. That knowledge doesn't get written down during a rotation; it accumulates in the person.
Where the noise actually comes from
In our experience building baselines across diverse observability stacks, alert noise typically concentrates in a few predictable patterns. Static thresholds are the most common source: a rule written 18 months ago when average CPU was 40% becomes noise at 70% average load as services grow. Nobody updates it because the alert has been firing for so long that nobody investigates it anymore.
Post-deploy transients are the second biggest source. Services spike on restart, warm-up periods are noisy, and any alert rule that doesn't account for recent deployment events will fire reliably during the exact windows when engineers are already paying close attention. This is the worst possible form of alert: it fires when you least need distraction, and it is almost always safe to ignore.
Correlated noise is the third pattern. A single root cause, say a slow downstream database, produces alerts across 6 different services simultaneously. Alertmanager can group these, but only if the grouping rules are configured correctly for the specific topology. Without grouping, a single real event generates a page storm that desensitizes engineers to the entire class of alert.
The fix is not fewer alert rules
The intuitive response to alert fatigue is to delete alert rules. This is sometimes right and often wrong. The problem is usually not that the rule shouldn't exist, but that the threshold is static, the context is missing, or the correlation is unresolved.
An alert rule for "CPU above 85% for 5 minutes" is reasonable. The same rule written without baseline context, without awareness of post-deploy windows, and without correlation to adjacent services will fire as noise constantly in a busy microservice environment. Deleting the rule removes important coverage. What you need instead is a detection layer that can evaluate the rule's firing in context: is this consistent with normal load variation for this service at this time of week? Did a deploy just happen? Are 4 other services showing correlated symptoms that suggest a shared upstream cause?
That contextual evaluation is exactly what ObsrvHQ does before deciding whether to escalate an alert to a human. The rule still fires internally. The question we're answering is whether this particular firing instance represents a situation that needs a human's attention right now.
Measuring progress
If you want to track whether your alert hygiene is improving, the most useful metric is not alert volume. It is the ratio of alerts that required human action to total alerts that fired. A high-noise environment might page an engineer 120 times in a week, of which 12 require real action. A 10% actionable rate. A healthy environment might page 30 times with 22 requiring action. A 73% actionable rate, dramatically less total noise even though the raw number looks lower.
Secondary metrics worth tracking: mean time from alert fire to engineer acknowledgment (a leading indicator of desensitization), rotation decline rate (how often engineers decline on-call shifts), and incident-to-postmortem ratio (low ratios often indicate incidents going undetected or unlogged because the engineer assumed it was noise).
None of these metrics are hard to collect. Most teams simply don't instrument them because alert quality is treated as a constant rather than a managed variable. It shouldn't be.