Skip to content

Silencing a noisy Datadog monitor during investigation

When a known issue is firing repeated pages and a fix is in flight, the engineer mutes the specific Datadog monitor scope for a fixed 30-minute window. The mute reason links the active PagerDuty incident, never widens to the whole monitor and never runs indefinitely.

Category
Tags
datadogmutemonitor-noisescopingauto-expiry
What and why
The observed behaviour and the reasoning behind it.
Behaviour
Reasoning
Cause and effect
What initiates this pattern and what it produces.
Trigger
Outcome
Standard operating procedure
Step-by-step instructions to reproduce this pattern.
1

Datadog

On the noisy monitor's detail page, click 'Mute Monitor' in the top right.

Use the Mute button on the monitor detail page rather than editing the monitor's settings. The Mute button is designed to expire automatically. Edits to the monitor itself are easy to forget and silently degrade the team's alert coverage long after the incident is over.

Expected: A mute dialog opens with fields for scope, duration and reason.

2

Datadog

Set the scope to the specific service and environment, not All scopes.

The default scope is everything the monitor watches, which usually includes staging, dev or other environments that share the monitor query. Type in the targeted scope tags exactly. Example: service:api-gateway env:prod. This keeps the staging signal alive even while production is muted.

Expected: The scope field shows the targeted tags rather than 'All'.

3

Datadog

Set the duration to 30 minutes.

Never mute indefinitely. The 30 minute floor is long enough to ship the fix and verify it, short enough that an unfixed regression will page again. If the fix needs longer, re-mute when the first window expires rather than extending up front. The act of re-muting is what keeps the team honest about whether the fix is actually progressing.

Expected: The duration is set to 30 minutes and an end time is shown.

4

Datadog

Type a mute reason that includes the PagerDuty incident ID.

Example: 'Muting for PD-PT4ZXKR, fix in flight via api-gateway #4901.' The PD ID gives whoever opens this monitor next an audit trail back to the incident. Without it, they will see only that the monitor is muted with no indication of why or for how long the issue is known.

Expected: The mute reason field contains the PD incident ID and a one line summary.

5

Datadog

Click Confirm to apply the mute.

Verify the monitor's status changes to Muted in the top right and that the muted scope is listed below the monitor name. If the mute did not register the monitor will continue to fire pages, which is the failure mode this step is guarding against.

Expected: The monitor shows Muted status with the scope and the auto-expiry time visible.

Related patterns
How this pattern connects to other patterns in the library.
Supporting actions
Actions that provide evidence for this pattern.
Muted api-gateway prod monitor for 30m, linked PD-PT4ZXKR
Silenced checkout-service env:prod scope, fix in flight
Re-muted billing-worker monitor after first 30m window expired
Set 30m mute on payments-service monitor with PD-Q9HK22N reason
Metadata
Timestamps and identifiers.
EvidenceObserved 14 times across 4 connections
ApplicationsDatadog, PagerDuty
First seen11 Feb 2026, 16:33
Last seen4 May 2026, 20:17
Questions

Frequently asked questions

Speak to the founder

Henry Denton, founder of FusedFrames

Get a demo. Watch a live capture, then an AI agent query the result.

Ask anything. Pricing, security or integrating with your stack.

No purchase obligation

Start capturing

Record in minutes. Install once and work as normal.

Plug AI agents in. One API call from any AI agent stack.

Refund on unused credits if you cancel