Quality assurance · Production

How Facebook leverages Large Language Models to understand user bug reports and guide fundamental improvements

The problem

User-filed bug reports arrive in unstructured free-text at massive scale, making analysis with traditional methods resource-intensive, hard to scale, and too slow to generate timely insights.

First attempt

Human reviewers and traditional machine learning models were the prior approach; human review was resource-intensive and hard to scale, while ML models fell short on unstructured free-text data.

Workflow diagram · grounded in source
1
User files bug report
trigger
“the reports submitted by users through an in-app mechanism, such as shaking the phone. It usually contains text descriptions”
2
LLM classification at scale
ai_action
“We developed an LLM-based classification system that assigns each report to a pre-defined category. This approach ensures structured understanding of unstructured feedback while enabling daily monitoring through integrated dashboards. It…”
3
Root cause rationalization
ai_action
“LLMs could go beyond classification. They can also be effective at 'rationalization' and answering "why are users experiencing issues", helping to find root causes during outages”
4
Dashboard monitoring and alerting
output
“We conduct weekly reporting and trend monitoring through our LLM-powered dashboards to track shifts in user complaints and identify emerging patterns. Our dashboard has data quality checks and threshold monitors set up to alert us to pot…”
5
Cross-functional product action
human_review
“we collaborate with a wide range of cross-functional teams — including Engineering, Product Management, User Experience Research, etc. — to identify system inefficiencies and build the solutions”
Reported outcome

The LLM-based system immediately detected major outages via real-time complaint pattern recognition, caught less-visible bugs earlier, and reduced topline bug reports by double digits over the last few months, with the methodology demonstrating measurable positive impacts throughout the user journey.

Reported metrics
Topline bug reports reduceddouble digits
Reported stack
Llama
Source
https://medium.com/@AnalyticsAtMeta/how-facebook-leverages-large-language-models-to-understand-user-bug-reports-and-guide-fundamental-70ab26475850
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM-based system immediately detected major outages via real-time complaint pattern recognition, caught less-visible bugs earlier, and reduced topline bug reports by double digits over the last few months, with th…

What tools did this team use?

Llama.

What results were reported?

Topline bug reports reduced: double digits (source-reported, not independently verified).

What failed first in this deployment?

Human reviewers and traditional machine learning models were the prior approach; human review was resource-intensive and hard to scale, while ML models fell short on unstructured free-text data.

How is this quality assurance AI workflow structured?

User files bug report → LLM classification at scale → Root cause rationalization → Dashboard monitoring and alerting → Cross-functional product action.

Related quality assurance cases