Incident management · Production

Meta builds AI-assisted root cause analysis to streamline incident investigations

The problem

Investigating issues in systems dependent on monolithic repositories is complex and time-consuming because thousands of code changes across many teams must be searched, and responders must rapidly build context on what is broken and who is impacted.

Workflow diagram · grounded in source

Investigation created

trigger

“identifying root causes for investigations at their creation time”

Heuristic retrieval narrows candidates

ai_action

“a novel heuristics-based retriever that is capable of reducing the search space from thousands of changes to a few hundred without significant reduction in accuracy using, for example., code and directory ownership or exploring the runti…”

LLM ranker selects top five

ai_action

“The ranker system uses a Llama model to further reduce the search space from hundreds of potential code changes to a list of the top five. We explored different ranking algorithms and prompting scenarios and found that ranking through el…”

Top candidates surfaced to responder

output

“42% of these investigations had the root cause in the top five suggested code changes”

Closed feedback loop and validation

feedback_loop

“we ensure that all employee-facing features prioritize closed feedback loops and explainability of results. This strategy ensures that responders can independently reproduce the results generated by our systems to validate their results”

Reported outcome

The AI-assisted root cause analysis system achieves 42% accuracy in surfacing the true root cause within its top five suggested code changes at investigation creation time for the web monorepo, reducing the effort and time needed to isolate root causes.

Reported metrics

Root cause identification accuracy42%

Investigation effort and timereduce effort and time needed to root cause an investigation significantly

Reported stack

LlamaLlama 2Hawkeye

Source

https://engineering.fb.com/2024/06/24/data-infrastructure/leveraging-ai-for-efficient-incident-response/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The AI-assisted root cause analysis system achieves 42% accuracy in surfacing the true root cause within its top five suggested code changes at investigation creation time for the web monorepo, reducing the effort and…

What tools did this team use?

Llama, Llama 2, Hawkeye.

What results were reported?

Root cause identification accuracy: 42%; Investigation effort and time: reduce effort and time needed to root cause an investigation significantly (source-reported, not independently verified).

How is this incident management AI workflow structured?

Investigation created → Heuristic retrieval narrows candidates → LLM ranker selects top five → Top candidates surfaced to responder → Closed feedback loop and validation.