Mean time to detect (MTTD) is one of the canonical SRE metrics, but the way most teams try to reduce it focuses on the wrong variable. The common approach is to add more alert rules, lower thresholds, and increase the number of signals that can trigger a page. The theory is that more coverage equals faster detection.
In practice, this often increases MTTD instead of reducing it. More alerts means more noise. More noise means engineers take longer to identify which alert represents the real problem. The signal you need to act on is buried in a stack of seven simultaneous pages, and the cognitive work of correlating them manually adds 10-15 minutes to detection time that would not exist if the correlation had been done before the page.
The manual correlation tax
When a distributed system incident starts, it typically produces a cascade. A database starts responding slowly. Three services that depend on it start showing elevated latency. Their error rates climb. SLO burn rate alerts fire. Redis connection timeouts begin appearing. A background job queue starts backing up. By the time an engineer is looking at PagerDuty, there may be 6-10 distinct alerts firing across different services, each with its own runbook link and each implying a different investigation path.
The engineer's actual first task is not to follow any of those runbooks. It is to mentally correlate the alerts and form a hypothesis about what is causing them. This takes time. Not because engineers are slow, but because the human brain is doing computational work that scales with the number of alerts: parsing timestamps to determine which fired first, cross-referencing service dependency maps that may or may not be accurate, and filtering out alerts that are clearly secondary effects.
In a noisy environment, this correlation work is harder because the engineer cannot assume all currently-firing alerts are related. Some may be independent noise from unrelated services that happened to breach thresholds at the same time. Distinguishing "this alert is a secondary effect of the database issue" from "this alert is unrelated noise from the auth service CPU rule" requires domain knowledge, context, and time.
Instrumenting this mental process is difficult, but the detection delay it introduces is real and measurable in postmortem timelines. A pattern we see consistently: the first alert fires at T+0 (relative to incident start). The engineer gets paged at T+2-3 minutes. The correct runbook is opened at T+12-18 minutes, after the correlation work is complete. MTTD is measured as T+12-18, but the detection latency relative to the first alert was T+0. The gap is the manual correlation tax.
Correlation at the detection layer
The alternative is to do the correlation work in the alerting infrastructure rather than in the engineer's head. This is what detection-layer correlation means. Instead of presenting an engineer with 8 independent alerts and expecting them to synthesize a coherent picture, you present them with 1 correlated incident alert that already describes the likely root cause area and lists the secondary symptoms.
For this to work, the detection system needs to understand two things: service topology (which services depend on which) and temporal correlation (which alerts tend to fire together as a group when a common cause is present).
Service topology can come from several sources: Kubernetes service labels and namespace relationships, service mesh telemetry, distributed tracing infrastructure, or explicit dependency declarations. ObsrvHQ ingests topology metadata from these sources and uses it to build a dependency graph that informs correlation logic.
Temporal correlation is learned from historical alert patterns. If alerts A, B, C, and D have historically fired together within a 3-5 minute window in 80-90% of cases when they fire, they almost certainly represent a correlated event pattern. When they fire again, grouping them as a single correlated incident alert is more informative than presenting them separately.
What "correlated incident alert" looks like in practice
Instead of receiving:
- PagerDuty incident: "payments-api error rate > 5%" (from alert rule)
- PagerDuty incident: "order-service latency p99 > 2s" (from alert rule)
- PagerDuty incident: "inventory-api error rate > 5%" (from alert rule)
- PagerDuty incident: "postgres connection pool utilization > 85%" (from alert rule)
The engineer receives:
- PagerDuty incident: "Correlated degradation across 4 services: postgres connection pool contention probable root cause (fired first at 02:14:32). Secondary effects: payments-api, order-service, inventory-api all showing downstream latency and error rate elevation. Correlation confidence: high."
The engineer's first action is now to check the Postgres instance, not to spend 10 minutes figuring out that Postgres is the common dependency. MTTD drops by the duration of that correlation work.
The topology-independent fallback
Not every incident follows a neat upstream-downstream topology. Some incidents are cross-cutting: a misconfigured load balancer, a network partition, a certificate expiry that affects multiple unrelated services. These don't have a single clear upstream root cause in the dependency graph.
For these cases, temporal correlation alone is still valuable. If 6 unrelated services all start firing alerts within a 60-second window, something shared is probably the cause even if the dependency graph doesn't reveal it. The most common shared dependencies that don't show up as service-level dependencies are network infrastructure, DNS, container runtime, and shared storage volumes.
Presenting these as a temporally correlated group, even without a root cause hypothesis, still compresses the engineer's cognitive work. They know immediately that the alerts are probably related, which eliminates the hypothesis "these are all independent coincidences" from the investigation tree. That alone meaningfully reduces MTTD.
Correlation is not a substitute for runbooks
It is worth being clear about limits here. Correlation-aware detection tells you what is probably happening and roughly where to look. It does not replace investigation. A correlated incident alert pointing to Postgres connection pool contention still requires an engineer to check what is consuming the connections, whether there is a query pattern change, whether connection limits were recently lowered, and whether this is a transient spike or a sustained condition. None of that is automated by correlation.
What correlation does is dramatically improve the starting point for that investigation. Instead of starting with "I have 8 alerts, which one matters," the engineer starts with "here is the likely root cause, here are the affected services." The investigation depth required doesn't change. The waste at the front end of the investigation, where the engineer would otherwise be doing correlation manually, is eliminated.
The cumulative effect on MTTD across a team over months is measurable. In well-correlated incident data, we see detection latency (from first alert to first relevant runbook action) compress from 12-20 minutes to 4-8 minutes for incidents where correlated alerts fired. That compression comes entirely from eliminating manual correlation work, not from adding coverage or lowering thresholds.