If your team uses Datadog Monitors as its primary alerting layer, the most common first question about adding ObsrvHQ is: "What do I have to change in Datadog?" The short answer is nothing. You don't modify monitor configurations, thresholds, or notification policies. You add ObsrvHQ as a webhook recipient alongside whatever notification channels you already have, and ObsrvHQ handles suppression from that position.
This guide walks through the complete setup. The integration takes roughly 20-30 minutes for a typical Datadog account. The steps below assume you have admin access to both Datadog and your ObsrvHQ account.
How the integration works conceptually
Datadog Monitors evaluate metric queries on a configured schedule and transition between OK, Alert, Warn, and No Data states. When a monitor transitions to Alert state, it sends notifications to configured channels: PagerDuty, Slack, email, or generic webhooks.
In the standard setup, that notification goes directly from Datadog to PagerDuty (or your incident management system), which creates an incident and pages the on-call engineer. There is no evaluation step between "monitor fires" and "engineer gets paged."
With ObsrvHQ in the path, you add a webhook notification to ObsrvHQ alongside your existing PagerDuty notification. When a monitor fires, ObsrvHQ receives the notification payload, evaluates the alert in the context of its baseline model for the tagged service, and decides whether to forward it onward. The difference: ObsrvHQ can also intercept the PagerDuty path by acting as a relay rather than a parallel recipient, which gives it full control over what reaches PagerDuty. Both topologies are supported; the relay model gives more control and is recommended for teams with significant noise.
Step 1: Create a Datadog API key for ObsrvHQ
ObsrvHQ uses the Datadog API to read monitor metadata: monitor names, tags, thresholds, and the service tags associated with each monitor. This metadata is used to build baseline groupings and understand which service each alert belongs to.
In your Datadog account, navigate to Organization Settings, then API Keys. Create a new API key with the name "ObsrvHQ integration" or similar. Copy the key value. You will need it in Step 3.
The key requires read access to Monitors (specifically the monitors_read permission if you are using scoped keys). It does not need write access to Monitors or access to your infrastructure agent configurations. ObsrvHQ does not modify any Datadog resources.
Step 2: Add the ObsrvHQ webhook to your monitor notification policy
In Datadog, navigate to Monitors, then Notification Settings (or Manage Downtime depending on your Datadog plan version). You will add a global webhook integration that fires for all monitors, rather than editing each monitor individually.
Go to Integrations, then Webhooks. Create a new webhook with the following configuration:
- Name:
obsrvhq-suppression - URL: your ObsrvHQ webhook endpoint (found in your ObsrvHQ account under Integrations, then Datadog)
- Payload: leave as default JSON or use ObsrvHQ's suggested payload template, which is available in your ObsrvHQ integration docs
- Custom headers: include your ObsrvHQ API key as
X-ObsrvHQ-Key: your-api-key
After saving the webhook, add @webhook-obsrvhq-suppression to the notification message template of your monitors. You can do this in bulk using the Datadog Monitors API if you have many monitors. The Monitors API endpoint PUT /api/v1/monitor/{monitor_id} accepts a message field update that can append the webhook mention without changing other monitor configuration.
Step 3: Connect your Datadog API key in ObsrvHQ
In your ObsrvHQ dashboard, go to Integrations and select Datadog. Paste the API key you created in Step 1. ObsrvHQ will immediately attempt to read your monitor list and display a summary of how many monitors were discovered and how many have identifiable service tags.
Service tag detection is important for baseline quality. ObsrvHQ looks for a service: tag on each monitor to associate it with a service in the baseline model. If your monitors use different tagging conventions (for example, app: or component:), you can configure the tag mapping in the ObsrvHQ integration settings. Monitors without a recognized service tag are still evaluated for suppression but use a less specific baseline model.
Step 4: Verify the integration and set the operating mode
Before suppression is active, ObsrvHQ runs in observe-only mode for a configurable period (default 7 days). During this period, it receives all monitor firing events, logs what it would have suppressed, and makes those shadow decisions available in the ObsrvHQ dashboard under Review Queue. This lets you evaluate suppression accuracy before it affects your PagerDuty notifications.
In observe-only mode, all alerts still reach PagerDuty as normal. Nothing changes for your on-call engineers. You can review the Review Queue in ObsrvHQ and verify that the suppression decisions look reasonable before activating suppression.
After the observation period, you activate suppression in the ObsrvHQ integration settings. At that point, alerts that ObsrvHQ classifies as noise in context are no longer forwarded to PagerDuty. They remain in the ObsrvHQ Review Queue where they can be reviewed, and any suppression decision can be manually overridden to force a PagerDuty notification if needed.
A note on Datadog Composite Monitors
Datadog Composite Monitors combine multiple monitors using boolean logic, which allows threshold combinations like "fire only if monitor A is in Alert AND monitor B is in Alert." This is Datadog's native approach to some of the correlation logic that ObsrvHQ handles at the suppression layer.
We are not saying Composite Monitors are redundant once you have ObsrvHQ. They serve a different function: they define a specific condition that must be true before an alert is considered active at all. ObsrvHQ evaluates that active alert in context after the Composite condition is met. The two mechanisms compose. A team that already uses Composite Monitors for known alert correlations should keep them; ObsrvHQ adds value on top by handling the dynamic contextual suppression that static Composite rules cannot express.
Troubleshooting common setup issues
The most common issue during setup is monitors not sending webhook notifications to ObsrvHQ. This usually happens because the @webhook-obsrvhq-suppression mention was added to only some monitors, not all. The Datadog Monitors API bulk update approach mentioned in Step 2 avoids this by programmatically ensuring the webhook mention is present in every monitor's message field.
The second common issue is service tag mismatches. If ObsrvHQ reports that a high percentage of monitors have no recognized service tag, check your Datadog monitor tagging conventions and configure the tag mapping in ObsrvHQ accordingly. Monitors without service tags still get evaluated, but with less contextual precision.
If you run into issues during setup, the ObsrvHQ integration diagnostics page shows the last 50 webhook events received and their processing status, which makes it straightforward to identify where in the flow a monitor's events are or are not appearing.