Incident management · Production

How Microsoft Security used hybrid ML and LLM to synthesize Post Incident Reviews at scale

The problem

Microsoft's Security team had large volumes of unstructured Post Incident Reviews with no scalable way to extract actionable insights, causing recurring themes to go undetected and thorough reviews to be deprioritized as analysts shifted focus to the next urgent incident.

First attempt

Pure ML approaches like topic modeling lacked the contextual depth to distinguish meaningful findings from passing mentions. Pure LLM end-to-end approaches were limited by context windows, cost, and risk of overly general or inaccurate output. Early prompt attempts produced unhelpfully vague summaries.

Workflow diagram · grounded in source
1
PIR data collection and prep
trigger
“We gathered a large set of past PIR documents from recent years. This involved working with our incident response teams to obtain reports and scrub highly sensitive details.”
2
Text embedding generation
ai_action
“We leveraged the Azure OpenAI Service to generate text embeddings for each PIR or each finding within a PIR. An embedding is like a numerical fingerprint of a piece of text that captures its semantic essence.”
3
K-means clustering
ai_action
“we applied clustering algorithms to group them. We started with k-means clustering (a well-known algorithm that groups data into k clusters based on distance in that vector space). But how to choose k (the number of clusters)? We employe…”
4
Expert cluster validation
human_review
“we involved our security experts to double-check: Do these clusters of incidents actually feel coherent and meaningful? This human validation was important.”
5
LLM cluster summarization
ai_action
“use an LLM (via the Azure OpenAI Service) to read the incidents in one cluster and produce a concise summary of that cluster's theme and key points”
6
Cluster summary output
output
“The output we requested was a paragraph or two, plus a short title for the cluster (e.g., "Misconfigured Access Controls" as a title, followed by explanation).”
7
New finding classification
ai_action
“every new PIR finding is embedded using the same model and then classified by comparing its embedding to the centroids of existing clusters. If the finding is close to a cluster, it is assigned accordingly; otherwise, it is flagged as po…”
8
Drift monitoring and retraining
feedback_loop
“the system can monitor for drift — such as an increase in outliers or changes in cluster cohesion. When drift is detected, we can re-run the clustering and summarization steps on the dataset to refresh clusters and update insights.”
Reported outcome

The hybrid pipeline significantly reduced the manual effort required to synthesize PIRs, enabling analysts to shift time toward higher-value validation and follow-up.
Teams gained a concise digest of recurring themes with remediation directions, improved cross-team knowledge sharing, and more decision-focused security review meetings.

Reported metrics
PIR synthesis effortcan now be produced much more quickly with AI assistance
Analyst time on repetitive summarizationshifted analyst time away from repetitive summarization toward higher-value validation
Review meeting efficiencystart from a synthesized view of themes rather than walking through individual incidents
Reported stack
Azure OpenAI Servicek-meansscikit-learnLLMsSLMs
Source
https://medium.com/data-science-at-microsoft/from-incidents-to-insights-how-hybrid-ai-supercharged-our-security-reviews-8672e0906cd1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The hybrid pipeline significantly reduced the manual effort required to synthesize PIRs, enabling analysts to shift time toward higher-value validation and follow-up.

What tools did this team use?

Azure OpenAI Service, k-means, scikit-learn, LLMs, SLMs.

What results were reported?

PIR synthesis effort: can now be produced much more quickly with AI assistance; Analyst time on repetitive summarization: shifted analyst time away from repetitive summarization toward higher-value validation; Review meeting efficiency: start from a synthesized view of themes rather than walking through individual incidents (source-reported, not independently verified).

What failed first in this deployment?

Pure ML approaches like topic modeling lacked the contextual depth to distinguish meaningful findings from passing mentions.

How is this incident management AI workflow structured?

PIR data collection and prep → Text embedding generation → K-means clustering → Expert cluster validation → LLM cluster summarization → Cluster summary output → New finding classification → Drift monitoring and retraining.

Related incident management cases