Quality assurance · Production

Meta Reality Labs builds Llama-powered RAG tool to extract product insights from customer feedback

The problem

Customer feedback data at Meta Reality Labs was underutilized despite its volume because the raw data was hindered by noise, bias, and lack of structure, making it difficult to extract meaningful insights.

Workflow diagram · grounded in source
1
Gather raw customer feedback
trigger
“Gather Raw Feedback: All customer feedback is assembled in their most granular and raw form”
2
Contextualize and summarize feedback
ai_action
“Contextualize Feedback: Join any pieces of raw feedback into a contextualized piece of feedback (e.g. elements of a "conversation" as depicted in Figure 2) — and then summarize it”
3
Embed feedback as vectors
ai_action
“Embed Feedback: Generate dense vector representations of the contextualized feedback, commonly referred to as embeddings. Embeddings capture the essence of meaning and context in a body of text, transforming words or phrases into vectors…”
4
Similarity search and retrieval
ai_action
“At runtime, the user's prompt is matched with relevant customer feedback by comparing its embedding to stored feedback embeddings using cosine similarity, retrieving the top-N most similar matches for consideration”
5
LLM generates response
ai_action
“Generate Response to User Prompt: The LLM generates a response to the user's prompt based on the contextually similar customer feedback retrieved”
6
Bug clustering and deduplication
output
“we're clustering bugs into natural groupings based on their descriptions and stack trace. Mapping bug reports to higher-level categories is helping prioritize and address the most critical areas reported by our users.”
7
QA feedback summarized for executives
output
“This data is being summarized in executive reporting, condensing hours of manual effort into minutes”
Reported outcome

Meta Reality Labs developed a self-service AI tool powered by Llama and RAG that enables analysts to query customer feedback for product insights, deduplicate bug reports, and condense hours of manual QA reporting effort into minutes.

Reported metrics
manual effort for QA executive reportingcondensing hours of manual effort into minutes
Reported stack
LlamaRAGYouTube
Source
https://medium.com/@AnalyticsAtMeta/harnessing-the-power-of-customer-feedback-unleashing-metas-llama-4-llms-in-product-analytics-2d3a9cfd5805
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Meta Reality Labs developed a self-service AI tool powered by Llama and RAG that enables analysts to query customer feedback for product insights, deduplicate bug reports, and condense hours of manual QA reporting eff…

What tools did this team use?

Llama, RAG, YouTube.

What results were reported?

manual effort for QA executive reporting: condensing hours of manual effort into minutes (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Gather raw customer feedback → Contextualize and summarize feedback → Embed feedback as vectors → Similarity search and retrieval → LLM generates response → Bug clustering and deduplication → QA feedback summarized for executives.

Related quality assurance cases