It support · Production

Uber improves Genie on-call copilot answer quality by 27% with Enhanced Agentic-RAG

The problem

Genie's traditional RAG pipeline over 40+ internal engineering security and privacy policy documents produced responses with significant accuracy gaps—many answers were incomplete, inaccurate, or failed to retrieve relevant information—preventing SME approval for broad deployment across Slack channels.

First attempt

Traditional PDF loaders (SimpleDirectoryLoader from LlamaIndex and PyPDFLoader from LangChain) failed to capture structured text and table formatting, fragmenting table cells from their row/column context and breaking downstream chunking and retrieval. Standard RAG accuracy improvements plateaued quickly and each evaluation cycle required weeks of SME bandwidth.

Workflow diagram · grounded in source
1
LLM-powered document enrichment
ai_action
“we built a custom Google document loader using the Google® Python API, extracting paragraphs, tables, and the table of contents recursively. For tables and structured text like bullet points, we integrated an LLM-powered enrichment proce…”
2
Metadata enrichment with summaries and FAQs
ai_action
“Leveraging the remarkable capabilities of LLMs to summarize large documents, we incorporated document summaries, a set of FAQs, and relevant keywords into the metadata.”
3
User query arrives in Slack
trigger
“designed to provide real-time support for thousands of queries across multiple help channels in Slack”
4
Query Optimizer refines query
ai_action
“Query Optimizer refines the query when it lacks context or is ambiguous. It also breaks down complex queries into multiple simpler queries for better retrieval.”
5
Source Identifier narrows document scope
routing
“Source Identifier then processes the optimized query to narrow down the subset of policy documents most likely to contain relevant answers. To achieve this, both agents use the document list artifact (titles, summaries, and FAQs) fetched…”
6
Hybrid vector and BM25 retrieval
ai_action
“we introduced an additional BM25-based retriever alongside traditional vector search. This retriever fetches the most relevant document chunks using enriched metadata, which includes summaries, FAQs, and keywords for each chunk. The fina…”
7
Post-Processor Agent structures context
ai_action
“The Post-Processor Agent performs two key tasks: de-duplication of retrieved document chunks and structuring the context based on the positional order of chunks within the original documents.”
8
Answer delivered via Slack
output
“The generated answer is then shared with the user through the Slack® interface.”
9
LLM-as-Judge automated evaluation
feedback_loop
“LLM-as-Judge module scores responses on a 0-5 scale, with 5 being the highest quality. It also provides reasoning for its evaluations, enabling us to incorporate feedback into future experiments.”
Reported outcome

Transitioning to the Enhanced Agentic-RAG architecture increased the percentage of acceptable answers by a relative 27% and reduced incorrect advice by a relative 60%, while automated evaluation cut experiment cycle time from weeks to minutes and produced a measurable reduction in support load for on-call engineers and SMEs.

Reported metrics
Acceptable answersrelative 27%
Incorrect advicerelative 60%
Experiment evaluation timefrom weeks to minutes
support load for on-call engineers and SMEsmeasurable reduction in the support load
Reported stack
GenieLangChainLangGraphLangfxMichelangeloLlamaIndexLlamaParsePyMuPDFPdfPlumberBM25Google DocsSlack
Source
https://www.uber.com/en-NL/blog/enhanced-agentic-rag/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Transitioning to the Enhanced Agentic-RAG architecture increased the percentage of acceptable answers by a relative 27% and reduced incorrect advice by a relative 60%, while automated evaluation cut experiment cycle t…

What tools did this team use?

Genie, LangChain, LangGraph, Langfx, Michelangelo, LlamaIndex, LlamaParse, PyMuPDF, PdfPlumber, BM25.

What results were reported?

Acceptable answers: relative 27%; Incorrect advice: relative 60%; Experiment evaluation time: from weeks to minutes; support load for on-call engineers and SMEs: measurable reduction in the support load (source-reported, not independently verified).

What failed first in this deployment?

Traditional PDF loaders (SimpleDirectoryLoader from LlamaIndex and PyPDFLoader from LangChain) failed to capture structured text and table formatting, fragmenting table cells from their row/column context and breaking…

How is this it support AI workflow structured?

LLM-powered document enrichment → Metadata enrichment with summaries and FAQs → User query arrives in Slack → Query Optimizer refines query → Source Identifier narrows document scope → Hybrid vector and BM25 retrieval → Post-Processor Agent structures context → Answer delivered via Slack → LLM-as-Judge automated evaluation.

Related it support cases