Incident Management Automation: What Production Deployments Show

Yes — with time-collapse numbers engineering teams published themselves. Artemis Security customers went from a two-hour average investigation to under five minutes, with the backlog described as simply disappearing. Agoda's security operation cut alert analysis from 20–40 minutes to under five across 400-plus alerts per fortnight at 97%-plus agreement with human analysts, runs phishing classification at over 99% precision in under 25 seconds, and drafts incident reports in under ten minutes that previously took five to seven hours — with a 30-minute human review keeping factual accuracy above 95%. Wix's AirBot handles pipeline failures across 3,500-plus Airflow pipelines, saving 675 engineering hours a month — about four full-time engineers — at roughly $0.30 per AI interaction, with 15% of its candidate fix PRs merging fully automatically.

The learning loop got automated too: Zalando's pipeline transformed two years of postmortems into queryable infrastructure insights, cutting analysis from days to hours — and a quarter of subsequent datastore incidents were shielded by the automated change validation the insights produced. LinkedIn's playbook system gives coding agents organisational context for 1,000-plus engineers, cutting issue triage time around 70%.

The shape everywhere: the AI assembles and drafts at machine speed; the engineer decides. Nothing in this record automates the judgment — everything automates the twenty browser tabs it used to require.

Context engineering — this category's failures are unusually precise because the people failing were instrumenting themselves. The canonical mistake: early SRE agents that performed many tool calls and summarised all telemetry at once, so token counts scaled linearly with incident complexity, model performance degraded, and the agent identified the wrong root cause with full confidence. The published fixes are token surgery: incident.io cut its Investigations agent's latency fourfold by stripping reasoning fields and compressing verbose dashboard JSON — an 80%-plus input-token reduction — because during an incident, an 11-second suggestion loop is a suggestion nobody waits for.

Model selection has its own documented ladder. Zalando's postmortem pipeline tried a consumer notebook tool first: severe hallucinations, lost incident context, productivity reduced rather than improved. Small open-source models hallucinated up to 40% of the time; human curation pulled that under 15%; a large frontier model made it negligible, at roughly 30 seconds per postmortem. That ladder — tool, small model, curation, frontier model — is the honest cost-quality curve for anyone building on operational text.

And one meta-failure unique to this category: evaluation in isolation. Testing individual tools separately passed while the agent still failed, because the failures emerged from interactions between steps. Incident agents must be evaluated on whole investigations — which is precisely why the teams that succeeded built replayable evaluation platforms before trusting anything in an outage.

This is one of the most build-native records on the site — engineering organisations automating their own house, publishing the internals — and the honest answer is layered. The response backbone is bought: paging, escalation, and status infrastructure is mature vendor territory, and it carries this category's unique procurement criterion, stated in the record itself: chat-only incident point products collapse when primary systems fail, lacking the multi-channel redundancy and failover depth enterprise operations require. Your incident tooling cannot share fate with the systems it exists to save — evaluate every candidate, bought or built, on what still works when Slack is down, when the cloud region is down, when the thing that broke is the thing it runs on.

The intelligence layer above the backbone is where the record builds: investigation agents on Claude and Bedrock, pipeline-failure bots, postmortem miners — engineering-team projects with published playbooks, honest failure modes, and startlingly good unit economics (thirty cents an interaction at Wix). The prerequisites are real, though: replayable evaluation before production trust, context engineering as a first-class discipline, and someone owning the agent the way someone owns a service.

Decision inputs: buy the backbone on resilience and integration depth; build the investigation layer if you have platform engineers and incident volume enough to teach it — the record's playbooks lower that bar every quarter; and whichever path, drill the failure of the tooling itself. An incident system is the one product whose worst day is also yours.