From alert storm to signal in 48 hours
A technical walkthrough for SREs and platform engineers evaluating ObsrvHQ. Covers ingestion architecture, baseline construction, suppression logic, and what you configure versus what the system learns on its own.
Where ObsrvHQ sits in your stack
ObsrvHQ is a filter layer between your metrics source and your alertmanager, not a replacement for either. Your Prometheus scrape configs, your Grafana dashboards, your PagerDuty escalation policies, your on-call schedules: none of that changes. ObsrvHQ adds a suppress-or-page decision point before alerts reach the people.
How baselines tighten over time
The minimum learning window is 48 hours. During that time, every alert passes through to your alertmanager unchanged while ObsrvHQ builds per-weekday, per-hour baseline envelopes for every metric series it observes. No alerts are suppressed until the first window completes.
First baselines established from 48h of observed time-series data. Initial suppression begins with wider envelopes and conservative sensitivity to avoid false suppression while the model is young.
Full Mon-Sun patterns are now visible. Envelopes tighten to match observed intraday rhythms. Suppression accuracy improves substantially at this stage.
Monthly traffic cycles, end-of-month batch jobs, scheduled maintenance patterns: all factored in. Long-tail anomalies that only appear occasionally are now detectable against a more precise baseline.
obsrvhq: source: prometheus endpoint: http://prometheus:9090 rules_path: /etc/prometheus/rules learning: min_window_hours: 48 granularity: per_hour_per_weekday sensitivity: 1.5 # multiplier suppress: mode: within_envelope deploy_window_min: 15
The decision logic for each incoming alert
For each incoming alert, ObsrvHQ asks: does this metric value fall within the learned envelope for this specific hour and day-of-week? If yes, suppress. If no, page. The sensitivity multiplier controls how much deviation beyond the envelope boundary triggers a page.
Configurable sensitivity multiplier. Default 1.5x. Widen envelopes to page less; tighten to catch more deviations.