Static threshold alerting has a straightforward mental model: pick a number, fire when the metric crosses it. This model served us well when services were monolithic, traffic was predictable, and deployments happened once a week. The threshold encoded knowledge that was stable: CPU above 90% meant saturation, latency above 500ms meant something was wrong.
Dynamic baselines are not new as a concept. What's changed is the operational environment that makes them necessary. In 2025, a standard microservice platform might have 50-300 distinct services, each with its own traffic pattern, deployment cadence, and scaling behavior. A threshold that's appropriate for the authentication service at 2pm on a Thursday is probably wrong for the same service during a promo event on a Saturday, and definitely wrong for a completely different service that happens to share the same metric name.
Why static thresholds drift toward noise
When you write a threshold rule, you're encoding the normal behavior you've observed at that moment. Your service gets 400 req/s at peak, so you set an error rate threshold based on that load. Three months later, the service is getting 1200 req/s at peak because the product grew. Your threshold is still correct in percentage terms, but the absolute error count that triggered it has tripled. Or the team added horizontal scaling and now peak load spreads across more instances, making each individual instance's CPU look lower, but you still have the same CPU threshold that triggers if any single instance hits it.
The threshold hasn't changed. The service has. This is the drift problem: rules age out of accuracy as services evolve, but nobody has the bandwidth to audit them continuously. Most production environments we've analyzed have a significant fraction of alert rules that haven't been touched in over a year. For any service that's actively developed, that rule is probably stale.
What a dynamic baseline actually models
A dynamic baseline models the expected distribution of a metric over time, accounting for periodic patterns. For most services, the relevant periodicities are: time of day, day of week, and sometimes day of month (billing cycles, reporting runs). A latency baseline for a B2B SaaS service will capture that latency is typically higher on Monday mornings when users batch-process weekend work, and lower on Sunday afternoons. These are expected patterns, not incidents.
The baseline is represented as an envelope: upper and lower bounds on expected metric values at each point in time, with configurable confidence intervals. A measurement that falls outside the envelope is a candidate for alerting. One that falls inside is not, regardless of whether it would have crossed a static threshold.
This is meaningfully different from anomaly detection in the statistical sense. We're not trying to identify all anomalies, which would include spurious statistical outliers that don't matter operationally. We're trying to identify measurements that are both outside the expected range and sustained (not just a single-sample spike), or that are accompanied by correlated changes in other metrics (which suggests a real incident rather than measurement noise).
The time-to-baseline problem
One practical challenge with dynamic baselines is the cold start: how long do you need to observe before the baseline is reliable? The answer depends on which periodicities matter for your service.
For most services, 7-14 days of clean observational data captures the daily and weekly cycles well enough to produce actionable baselines. A service that's been running stably in production for months will reach this faster than a service that just launched, because historical metric data is available to seed the learning process.
The cold start problem is more acute for services with irregular traffic patterns: a batch processing service that runs only on weekends, or a reporting service with monthly peaks. For these, you need to observe at least 2-3 full cycles before the baseline has seen enough of the relevant pattern. In practice, this means the baseline for a monthly batch job might need 6-8 weeks before it's reliable. During that period, you stay in passthrough mode.
We handle this by tagging each metric baseline with a confidence score that reflects how many complete cycles the model has observed. Suppression only activates when the confidence score crosses a threshold. A brand-new service's metrics will all have low confidence scores; an established service's metrics will have high scores. You see this explicitly in the ObsrvHQ rule browser so you're never guessing which baselines are active.
Baseline drift detection: the other direction
The baseline needs to track the service as it evolves, but it can't be so responsive that it absorbs incidents into the normal range. This is the core tension in adaptive baseline design.
Consider a database service where query latency has been at 45ms p99 for months. Then an infrastructure issue causes it to spike to 300ms and stay there for 48 hours. An adaptive baseline that updates too fast would shift its envelope to include 300ms as normal, making the sustained degradation invisible. An adaptive baseline that updates too slowly won't capture legitimate performance improvements: if the team does a major query optimization and p99 drops to 20ms permanently, the old baseline's lower bound would keep generating false-positive "too low" anomalies that don't exist in practice.
The approach we use is asymmetric update velocity: baselines absorb gradual upward drift slowly (requiring multiple days of sustained high values before shifting the envelope up), but absorb downward drift faster (a lasting improvement is incorporated relatively quickly). This isn't a universal rule for all domains, it's specifically calibrated for operational metrics where sustained degradation is the thing you most want to catch.
There's also the case of intentional regime change, typically a major refactor or architectural change. In these situations, you want to explicitly reset the baseline rather than waiting for drift detection to catch up. ObsrvHQ supports a baseline reset operation that you can trigger manually or via CI/CD webhook. After a reset, the metric goes back into learning mode for a configurable window.
Dynamic baselines and SLOs together
One framing that helps teams adopt dynamic baselines is to think of them as a complement to SLO-based alerting rather than a replacement. SLO burn rate alerts are excellent for customer-visible reliability: if your 30-day error budget is burning too fast, you need to know. These alerts are intentionally coarse-grained and slow-to-fire; they're optimized for not missing important degradation over a longer time window.
Dynamic baselines work at a finer granularity: they catch things that might be developing into an SLO problem before the burn rate shows it. A latency increase that's still within SLO compliance today but trending the wrong way is exactly the kind of signal dynamic baselines capture. You get earlier warning with fewer false positives than static thresholds would produce at the same sensitivity level.
This is the combination we'd recommend for most platform teams: SLO burn rate alerts as your primary customer-impact signal, dynamic baseline anomaly detection as your early warning layer. The two don't compete; they operate at different time scales and with different precision tradeoffs.
Building vs. buying baseline infrastructure
A reasonable question: can you implement dynamic baselines yourself? Yes, technically. The math for time-series decomposition into trend, seasonality, and residual components (STL, Prophet, or simpler Holt-Winters exponential smoothing approaches) is well-documented and not exotic. Prometheus Recording Rules let you precompute baseline envelopes. Several teams do this.
The part that's harder to build yourself is the operational layer around the math: managing baseline state across hundreds of services, handling cold starts correctly, propagating drift detection reliably, and surfacing confidence scores in a way that's actionable. These are not research problems, but they are non-trivial engineering that doesn't directly deliver product value to your users. That's the build-vs-buy tradeoff your team needs to make explicitly.
We built ObsrvHQ specifically to avoid putting that operational overhead on platform teams. The goal is that you get the noise reduction without having to become experts in time-series forecasting infrastructure. Whether that tradeoff makes sense depends on your team's size and priorities.