It support · Production

Uber builds Enhanced Agentic-RAG (EAg-RAG) to raise Genie on-call copilot answer quality to near-human precision

The problem

Genie's initial RAG implementation, when tested against 100+ golden queries on Uber's engineering security and privacy policy documents, produced responses that were incomplete, inaccurate, or failed to retrieve relevant information — falling short of the standards needed for broader Slack channel deployment. PDF loaders also failed to preserve complex table formatting, degrading downstream chunking and retrieval quality.

First attempt

Traditional PDF loaders (SimpleDirectoryLoader from LlamaIndex and PyPDFLoader from LangChain) stripped table formatting, disconnecting cells from their row and column context and degrading chunking and retrieval. Experiments with newer loaders including PdfPlumber, PyMuPDF, and LlamaParse could not provide a universal solution. SME-driven evaluation also took weeks per experiment, stalling iteration with only marginal accuracy gains.

Workflow diagram · grounded in source
1
Enriched document ingestion
ai_action
“we built a custom Google document loader using the Google® Python API, extracting paragraphs, tables, and the table of contents recursively. For tables and structured text like bullet points, we integrated an LLM-powered enrichment proce…”
2
User query submitted via Slack
trigger
“provide real-time support for thousands of queries across multiple help channels in Slack®”
3
Query Optimizer refines query
ai_action
“Query Optimizer refines the query when it lacks context or is ambiguous. It also breaks down complex queries into multiple simpler queries for better retrieval.”
4
Source Identifier narrows document scope
routing
“Source Identifier then processes the optimized query to narrow down the subset of policy documents most likely to contain relevant answers. To achieve this, both agents use the document list artifact (titles, summaries, and FAQs) fetched…”
5
Hybrid vector + BM25 retrieval
ai_action
“we introduced an additional BM25-based retriever alongside traditional vector search. This retriever fetches the most relevant document chunks using enriched metadata, which includes summaries, FAQs, and keywords for each chunk. The fina…”
6
Post-Processor Agent structures context
validation
“The Post-Processor Agent performs two key tasks: de-duplication of retrieved document chunks and structuring the context based on the positional order of chunks within the original documents.”
7
Answer generated and delivered via Slack
output
“The generated answer is then shared with the user through the Slack® interface”
8
LLM-as-Judge automated evaluation
feedback_loop
“we use an LLM to assess chatbot responses (x) within a given context (C), producing structured scores, correctness labels, and AI-generated reasoning and feedback”
Reported outcome

By transitioning to an Enhanced Agentic-RAG (EAg-RAG) architecture, Uber increased the percentage of acceptable answers by a relative 27% and reduced incorrect advice by a relative 60%, enabling Genie to scale across multiple security and privacy Slack channels with a measurable reduction in on-call engineer and SME support load.

Reported metrics
Acceptable answers increaserelative 27%
Incorrect advice reductionrelative 60%
Experiment evaluation timereduced from weeks to minutes
SME and on-call engineer support loadmeasurable reduction
Reported stack
GenieLangChainLangGraphLlamaIndexLlamaParsePdfPlumberPyMuPDFMichelangeloLangfxhtml2textMarkdownifyGoogle DocsBM25Slack
Source
https://www.uber.com/en-GB/blog/enhanced-agentic-rag/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By transitioning to an Enhanced Agentic-RAG (EAg-RAG) architecture, Uber increased the percentage of acceptable answers by a relative 27% and reduced incorrect advice by a relative 60%, enabling Genie to scale across…

What tools did this team use?

Genie, LangChain, LangGraph, LlamaIndex, LlamaParse, PdfPlumber, PyMuPDF, Michelangelo, Langfx, html2text.

What results were reported?

Acceptable answers increase: relative 27%; Incorrect advice reduction: relative 60%; Experiment evaluation time: reduced from weeks to minutes; SME and on-call engineer support load: measurable reduction (source-reported, not independently verified).

What failed first in this deployment?

Traditional PDF loaders (SimpleDirectoryLoader from LlamaIndex and PyPDFLoader from LangChain) stripped table formatting, disconnecting cells from their row and column context and degrading chunking and retrieval.

How is this it support AI workflow structured?

Enriched document ingestion → User query submitted via Slack → Query Optimizer refines query → Source Identifier narrows document scope → Hybrid vector + BM25 retrieval → Post-Processor Agent structures context → Answer generated and delivered via Slack → LLM-as-Judge automated evaluation.

Related it support cases