Incident management · Production

Microsoft Research uses LLMs to recommend root cause and mitigation steps for cloud incidents

The problem

Hyperscale cloud services like Microsoft 365 face the challenge of quickly detecting incidents and performing root cause analysis and mitigation at scale, with significant engineering effort required for manual resolution.

Workflow diagram · grounded in source

Incident ticket created

trigger

“When an incident ticket is created, the author specifies a title for each incident created and describes any relevant details, such as error messages, anomalous behavior, and other details which might help with resolution.”

LLM processes incident text

ai_action

“We used the title and the summary of a given incident as the input for LLMs and generated root cause and mitigation steps”

On-call engineer evaluates recommendations

human_review

“we also interviewed the incident owners to evaluate the effectiveness of the generated recommendations. Overall, GPT-3.5 outperforms GPT-3 in a majority of the metrics. More than 70% of on-call engineers gave a rating of 3 out of 5 or be…”

Reported outcome

Fine-tuned GPT-3.5 substantially outperformed GPT-3 models, improving average lexical similarity by 45.5% for root cause generation and 131.3% for mitigation generation over zero-shot settings; more than 70% of on-call engineers found the recommendations useful.

Reported metrics

root cause generation improvement (fine-tuned GPT-3.5 vs zero-shot)45.5%

mitigation generation improvement (fine-tuned GPT-3.5 vs zero-shot)131.3%

GPT-3.5 gain over GPT-3 models for root cause taskat least 15.38%

GPT-3.5 gain over GPT-3 models for mitigation taskat least 11.9%

Show all 8 reported metrics

root cause generation improvement (fine-tuned GPT-3.5 vs zero-shot)45.5%

mitigation generation improvement (fine-tuned GPT-3.5 vs zero-shot)131.3%

GPT-3.5 gain over GPT-3 models for root cause taskat least 15.38%

GPT-3.5 gain over GPT-3 models for mitigation taskat least 11.9%

GPT-3.5 gain over GPT-3 for mitigation with root cause inputat least 11.16%

on-call engineer usefulness rating (3/5 or better)More than 70%

incidents in studymore than 40,000

services covered in studymore than 1000

Reported stack

GPT-3GPT-3.5IcM

Source

https://www.microsoft.com/en-us/research/blog/large-language-models-for-automatic-cloud-incident-management/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Fine-tuned GPT-3.5 substantially outperformed GPT-3 models, improving average lexical similarity by 45.5% for root cause generation and 131.3% for mitigation generation over zero-shot settings; more than 70% of on-cal…

What tools did this team use?

GPT-3, GPT-3.5, IcM.

What results were reported?

root cause generation improvement (fine-tuned GPT-3.5 vs zero-shot): 45.5%; mitigation generation improvement (fine-tuned GPT-3.5 vs zero-shot): 131.3%; GPT-3.5 gain over GPT-3 models for root cause task: at least 15.38%; GPT-3.5 gain over GPT-3 models for mitigation task: at least 11.9% (source-reported, not independently verified).

How is this incident management AI workflow structured?

Incident ticket created → LLM processes incident text → On-call engineer evaluates recommendations.