When we talk to platform teams about ObsrvHQ, the question that comes up most often is some version of: "Do we need this if we already have Alertmanager?" It is a fair question, and the honest answer requires understanding exactly what Alertmanager is designed to do, because it is not designed to do what we do.
Prometheus Alertmanager is a routing, grouping, and deduplication layer. It receives alert events that Prometheus has already decided to fire, and it determines how those events should be delivered: which receiver gets them (PagerDuty, Slack, email), how similar alerts should be grouped into a single notification, and whether a currently-firing alert has already been sent within a given window. It is excellent at all of these things.
What Alertmanager does not do is evaluate whether a given alert firing should result in a human being paged at all. That decision happens upstream, in the Prometheus rule evaluation engine, and Alertmanager has no visibility into it. If Prometheus decides to fire an alert, Alertmanager's job is to route it, not to question it.
Where the problem actually lives
Alert fatigue is not primarily a routing problem. It is a pre-routing problem. The noise exists before Alertmanager sees it. By the time an alert event reaches Alertmanager, the question of whether it should have fired at all has already been answered with "yes" by the rule evaluation engine, regardless of whether that answer is correct in the current context.
Consider the post-deploy noise scenario. A service restarts, warms up, and briefly exceeds CPU and memory thresholds during the warm-up period. Prometheus fires the CPU alert and the memory alert. Alertmanager groups them into a single notification and routes them to PagerDuty. An incident is created. The on-call engineer wakes up, opens the incident, sees it is 3 minutes post-deploy, marks it resolved, and goes back to sleep. This is working as designed, including the human-wakeup. There was no routing error. The routing was correct. The problem is that the alert should never have fired in the first place, because "CPU high during a post-deploy warm-up window" is a known non-incident pattern.
Alertmanager cannot suppress this because it doesn't have the context that a deploy just happened. Even if you use Alertmanager's inhibition rules carefully, inhibiting alerts requires defining the inhibition criteria in advance as a static rule, which cannot account for dynamic context like deployment timestamps or baseline comparison.
What ObsrvHQ adds to a Prometheus stack
ObsrvHQ operates between Prometheus rule evaluation and Alertmanager. When a Prometheus alert fires, we receive it via webhook before it reaches Alertmanager. We then evaluate that alert event against our baseline model for the relevant service: Is this firing consistent with the service's normal behavioral pattern at this time of day and week? Did a deployment happen in the past 10-15 minutes? Are other metrics in the service's correlated group showing consistent or inconsistent signals? Is this alert part of a pattern that has historically resolved without human intervention?
If the answers point toward "this is noise in context," we suppress the event and log it for review. If the answers are ambiguous or point toward a genuine anomaly, we pass the event to Alertmanager, which then handles routing and notification as it normally would.
The key point: Alertmanager's behavior doesn't change. Its routing rules, grouping configuration, and notification policies remain exactly as configured. We are not replacing any of that. We are inserting a context-evaluation layer upstream of it.
What they each do well
Alertmanager is well-designed for its problem. The routing DSL is expressive, the grouping logic is flexible, the receiver ecosystem is comprehensive. If you have complex routing requirements across multiple teams with different notification preferences, Alertmanager handles that cleanly. Its inhibition rules, when written correctly, handle some static suppression cases. Its silencing mechanism is useful for planned maintenance windows.
What Alertmanager cannot do: evaluate firing context dynamically, compare against historical baselines, detect correlated anomalies across multiple metrics, or learn from patterns of human dismissal. These are not omissions in Alertmanager's design. They are simply outside its scope.
ObsrvHQ's scope is exactly that gap. We are not a routing tool. We don't manage receivers or notification policies. We evaluate a single question: given everything we know about this service's historical behavior and the current system state, does this alert firing event represent a situation that requires human attention right now?
The architecture in practice
The integration pattern is straightforward. In your Prometheus alerting configuration, you add ObsrvHQ's webhook URL as an additional alerting endpoint alongside your existing Alertmanager endpoint. Alternatively, some teams route all Prometheus alerts to ObsrvHQ and have ObsrvHQ forward to Alertmanager selectively. Either topology works.
A typical configuration looks like this in terms of event flow:
- Prometheus rule evaluation fires an alert event
- Event is forwarded to ObsrvHQ webhook receiver
- ObsrvHQ evaluates baseline context (typically takes 50-200ms)
- If anomalous: event forwarded to Alertmanager for routing and notification
- If noise in context: event logged to ObsrvHQ review queue, not forwarded
- Alertmanager handles routing for forwarded events as configured
The review queue is important. Suppressed events don't disappear. They are logged with the suppression reason (e.g., "post-deploy window," "within baseline range," "correlated pattern consistent with normal traffic variation") and are available in the ObsrvHQ dashboard. This gives platform teams visibility into what is being suppressed and why, which is essential for building confidence in the system and for auditing suppression decisions when incidents are later discovered.
When you might not need ObsrvHQ
We are not saying every team with Alertmanager needs an additional layer. If your alert rules are already highly curated, your false positive rate is low, and your on-call team is not experiencing fatigue, Alertmanager handles your situation fine. The case for adding a context evaluation layer is strongest when your alert rule set has grown organically over multiple years, when your services have variable load patterns that make static thresholds noisy, or when post-deploy transient alerts are a significant contributor to on-call load.
The simplest test: if more than 30% of PagerDuty incidents in a given month are resolved with no action taken, you have a pre-routing problem that Alertmanager alone cannot fix. That is where ObsrvHQ earns its place in the stack.