It support · Production

Genie: Uber's generative AI on-call copilot saves 13,000 engineering hours via RAG over internal docs

The problem

Uber's internal engineering Slack support channels received around 45,000 questions per month, but high volumes and long response wait times reduced productivity for both users and on-call engineers; relevant answers were hard to find because documentation was fragmented across Engwiki, internal Stack Overflow, and other locations, leading users to ask the same questions repeatedly.

Workflow diagram · grounded in source
1
User posts question in Slack
trigger
“when a user posts a question in a Slack channel”
2
Internal docs scraped and indexed
integration
“we scrape internal data sources like Uber's internal wiki, Uber's internal Stack Overflow, engineering requirement documents, and create vectors from these data sources using an Open AI embedding model. Those embeddings get stored in a v…”
3
Query embedded and searched
ai_action
“the question gets translated to embeddings. The service searches for relevant embeddings related to the question in a vector database”
4
LLM generates cited response
ai_action
“We asked the LLM to only give answers from the various sub-contexts provided and return the source url to cite the answer. This seeks to provide a source URL for every answer it returns”
5
User feedback via Slack buttons
feedback_loop
“a Slack plugin picks it up and uses a specific Kafka topic to stream metrics into a Hive table with the feedback and all the relevant metadata. We later visualize these metrics in a dashboard”
6
Escalation to on-call support
human_review
“If Genie can't answer their questions, users can easily escalate the issue to on-call support”
7
Evaluation pipeline tunes RAG
feedback_loop
“We provide Genie users with the option to run custom evaluations. They can evaluate hallucinations, answer relevancy, or any other metric that they deem important for their use case. This evaluation can be used for better tuning of all t…”
Reported outcome

Since its September 2023 launch, Genie has expanded to 154 Slack channels, answered over 70,000 questions, achieved a 48.9% helpfulness rate, and saved an estimated 13,000 engineering hours.

Reported metrics
monthly questions on Slack channels45,000
Slack channels served by Genie154
Total questions answeredover 70,000
Helpfulness rate48.9%
Show all 5 reported metrics
monthly questions on Slack channels45,000
Slack channels served by Genie154
total questions answeredover 70,000
helpfulness rate48.9%
engineering hours saved13,000
Reported stack
SlackRAGApache SparkOpenAI embedding modellangchainTerrablobSiaKafkaHiveMichelangelo Gateway
Source
https://www.uber.com/us/en/blog/genie-ubers-gen-ai-on-call-copilot/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Since its September 2023 launch, Genie has expanded to 154 Slack channels, answered over 70,000 questions, achieved a 48.9% helpfulness rate, and saved an estimated 13,000 engineering hours.

What tools did this team use?

Slack, RAG, Apache Spark, OpenAI embedding model, langchain, Terrablob, Sia, Kafka, Hive, Michelangelo Gateway.

What results were reported?

monthly questions on Slack channels: 45,000; Slack channels served by Genie: 154; Total questions answered: over 70,000; Helpfulness rate: 48.9% (source-reported, not independently verified).

How is this it support AI workflow structured?

User posts question in Slack → Internal docs scraped and indexed → Query embedded and searched → LLM generates cited response → User feedback via Slack buttons → Escalation to on-call support → Evaluation pipeline tunes RAG.

Related it support cases