An alert rule is a snapshot of an assumption. When you write alert on error_rate > 0.5% for 5m, you're encoding a belief: that a 0.5% error rate for 5 continuous minutes is unusual for this service and warrants a page. That belief was probably accurate when you wrote the rule. It may not be accurate 6 weeks later.
The 30-day framing in the title is a rough heuristic, not a precise measurement. What we mean is that the conditions that make an alert rule accurate at creation time are often different from the conditions 30, 60, or 90 days later. Services evolve faster than rule sets do. And the gap between where the rule is and where the service has gone is what generates noise.
The three ways alert rules go stale
Not all staleness is the same. Understanding the mechanism helps you target the right fix.
The first mechanism is traffic growth. You wrote the rule when the service handled 100 req/s at peak. Now it handles 800 req/s. A 0.5% error rate at 100 req/s means 0.5 errors per second. At 800 req/s, the same percentage produces 4 errors per second. Whether that's more or less alarming depends on your error budget, your SLO, and what the errors actually are. But the rule doesn't know any of this; it only knows whether the percentage crossed the threshold. If you haven't updated the rule to account for the different operating scale, the threshold may be either too sensitive (firing on noise at higher volume) or not sensitive enough (permitting a level of errors that's actually burning your budget faster than intended).
The second mechanism is behavioral change from deploys. Your service ships new code regularly. Each significant deploy can shift the baseline behavior of the service: response time distributions change, memory usage patterns change, which code paths handle which request types changes. A rule written against the performance profile of v1.4 of your service is measuring the v2.1 of your service against v1.4 expectations. Sometimes the rule still works. Often it generates noise on patterns that are different from v1.4 but entirely normal for v2.1.
The third mechanism is seasonal and cyclical change. Most services have usage patterns that vary by time of day, day of week, and sometimes quarter or season. A rule that was calibrated during a normal-load period will fire more often during high-load periods, even if nothing is wrong. The inverse is also true: rules calibrated during a high-traffic promotion period may be too tolerant during normal periods, missing degradation that would have been caught by a tighter threshold.
Why teams don't fix this
The failure mode is well-understood. Most platform teams know their alert rules drift. The reason they don't fix it systematically is that rule maintenance competes with everything else on the team's backlog, and the cost of not maintaining rules is distributed and slow-accumulating rather than acute and obvious.
When a rule fires incorrectly, the immediate cost is one engineer's interrupted sleep and 20 minutes of investigation that goes nowhere. That's annoying but not catastrophic. The cost accumulates across hundreds of such pages over months, but it never generates a specific ticket or postmortem action item because no single page was the crisis. The work of maintaining rules has to be justified against this distributed, invisible cost, which is hard to make the case for in a sprint planning session where there are visible features to ship.
There's also a psychological dynamic. Rules that were written after incidents carry weight. "We don't have an alert for that" is something teams fear after missing a real failure, and it drives rule creation. But "we have an alert that's been misfiring for 8 months" doesn't drive the same urgency for removal, even though a misfiring rule that no one trusts may actually make the next real incident harder to detect (engineers learn to tune out rules that habitually fire noise).
The self-defeating alert: when noise cancels signal
The most serious consequence of stale alert rules isn't just wasted time on false positives. It's the erosion of signal quality for real incidents.
Consider a rotation where 70% of pages are noise. Engineers on that rotation develop a pattern: wake up, check the obvious symptoms quickly, acknowledge, go back to sleep. This is rational optimization for the common case. The common case is noise. But this optimization is dangerous when the 30% real incidents require the same initial workflow. The on-call engineer can't tell from the page alone which kind of incident they've received. If their conditioned response is to investigate quickly and dismiss, a real incident gets the same quick-dismiss treatment at 3am.
We're not saying engineers deliberately ignore incidents. We're saying that when the base rate of real incidents is low and the cost of fully investigating each page is high, the economically rational response is to develop faster triage heuristics. Those heuristics can fail when a real incident looks similar to the habitual noise. This is an emergent reliability risk that comes entirely from stale alerting, not from engineers being careless.
What alert rule maintenance actually requires
The teams with the least alert noise are not the ones that never add rules. They're the ones that maintain a regular review cycle and treat alert rules as code that needs to match the production environment.
A practical cadence: quarterly rule review for stable services, monthly for rapidly evolving ones. The review asks three questions per rule. Has this rule fired in the last review period? If so, what fraction of fires were real incidents vs. noise? If the noise fraction is above roughly 50%, the threshold needs adjusting, the rule needs SLO-budget context, or the rule should be removed.
The hard part is that this review requires accurate data on which pages were real and which were noise. If your on-call rotation doesn't have a consistent practice of tagging incident resolutions (real incident vs. noise vs. self-resolved), you can't do this analysis. Building that practice is a prerequisite for systematic rule maintenance, and it's a social norm before it's a tooling problem.
Where ObsrvHQ fits in the rule lifecycle
Baseline learning doesn't fix stale rules directly. It provides a noise suppression layer that reduces the immediate impact of staleness while the underlying rules age. A rule that would generate 15 false positives per week generates 2-3 observable notifications per week once suppression is active for that rule. The rule is still stale; it's still firing. You still need to fix or remove it. But the on-call team isn't getting paged for it in the interim.
The other thing baseline suppression does is make stale rules more visible. ObsrvHQ's rule browser shows fire rate and suppression rate per rule. A rule with a 90% suppression rate is a rule with a very high noise fraction. That's a direct indicator of a rule that needs attention: either the threshold is wrong for the current service behavior, or the rule is watching a metric that's no longer meaningful for this service, or the rule was never well-calibrated to begin with.
The goal isn't to keep suppressing indefinitely. The goal is to use the suppression window to do the rule maintenance work without sacrificing the on-call team's sleep quality in the meantime. Noise reduction and rule quality improvement are complementary, not competing. Fixing the rules at the source is still the right long-term outcome. The baseline layer buys you the time and clarity to do that work without the cost of the noise accumulating while you do.