Incident management · Production

PagerDuty's AI Data Engineering Team cuts on-call incidents by 30% with automated alert management

The problem

The on-call team faced high incident volumes with many non-actionable alerts requiring manual snoozing to monitor, and the load grew unsustainably as new services were continuously deployed.

Workflow diagram · grounded in source
1
Pull incidents insights report
trigger
“Pull an Insights >> Incidents report by team Include the created at, resolved at, acknowledged by, TTR, resolved by, auto-resolve columns Download the report to CSV”
2
Filter for unactionable incidents
validation
“Filter report to view potentially unactionable incidents - Incidents that were not acknowledged or resolved by a human - Incidents that auto-resolved within 2-5 minutes - Incidents for "staging" or "test" environments”
3
Implement configuration changes
integration
“Implement configuration changes and measure results”
4
Automated incident snoozing
routing
“Self-resolving incidents are automatically snoozed and only notify the on-call if they are not resolved within a certain period of time.”
5
Rule inheritance for new services
feedback_loop
“Newly provisioned services inherit previously configured rules to ensure known noisy alerts are deprioritized or suppressed.”
Reported outcome

After implementing configuration changes, the team cut on-call incidents by 30%, reduced mean time to acknowledge dramatically through better alert quality, and eliminated manual snoozing through automated rules.

Reported metrics
On-call incident reduction30%
Incident volumeSignificant reduction in incident volume
mean time to acknowledge (MTTA)dropped dramatically
High urgent incidentsFewer false alarms
Show all 5 reported metrics
on-call incident reduction30%
incident volumeSignificant reduction in incident volume
mean time to acknowledge (MTTA)dropped dramatically
high urgent incidentsFewer false alarms
manual snoozing workLess time spent snoozing
Reported stack
PagerDuty
Source
https://www.pagerduty.com/ops-guides/using-pd/ai-data-engineering-team/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After implementing configuration changes, the team cut on-call incidents by 30%, reduced mean time to acknowledge dramatically through better alert quality, and eliminated manual snoozing through automated rules.

What tools did this team use?

PagerDuty.

What results were reported?

On-call incident reduction: 30%; Incident volume: Significant reduction in incident volume; mean time to acknowledge (MTTA): dropped dramatically; High urgent incidents: Fewer false alarms (source-reported, not independently verified).

How is this incident management AI workflow structured?

Pull incidents insights report → Filter for unactionable incidents → Implement configuration changes → Automated incident snoozing → Rule inheritance for new services.

Related incident management cases