Back office ops · Production

QueryGPT: Uber builds a natural language to SQL system using LLMs and multi-agent architecture

The problem

Authoring SQL queries at Uber required deep knowledge of SQL syntax and internal data models and took around 10 minutes per query, creating a productivity bottleneck across approximately 1.2 million interactive queries per month.

First attempt

The initial version of QueryGPT used simple RAG over a small sample set and suffered declining accuracy as more tables were onboarded; simple similarity search between natural language prompts and SQL schemas returned irrelevant results, and large schemas exceeded the available LLM token limit.

Workflow diagram · grounded in source
1
User submits natural language prompt
trigger
“enabling users to generate SQL queries through natural language prompts”
2
Intent Agent classifies domain
ai_action
“Every incoming prompt from the user now first runs through an "intent" agent. The purpose of this intent agent is to map the user's question to one or more business domains/workspaces (and by extension a set of SQL samples and tables map…”
3
Table Agent selects tables
ai_action
“we added another LLM agent (Table Agent) that would pick the right tables and send those out to the user to either "ACK" or edit the given list and set the right tables”
4
User reviews and confirms tables
human_review
“The user would either select the "Looks Good" button or edit the existing list and modify the list of tables to be used by the LLM for query generation”
5
Column Prune Agent trims schemas
ai_action
“we implemented a "Column Prune" agent, wherein we use an LLM call to prune the irrelevant columns from the schemas we provided to the LLM. This change massively improved not just the token size and by extension the cost of each LLM call,…”
6
SQL query generated and returned
output
“The response would include an "SQL Query" and an "Explanation" of how the LLM generated the query”
7
Evaluation and continuous improvement
feedback_loop
“To track incremental improvements in QueryGPT's performance, we needed a standardized evaluation procedure. This enabled us to differentiate between repeated vs. anomalous shortcomings of the service and ensure algorithm changes were inc…”
Reported outcome

QueryGPT reduced SQL query authoring time from around 10 minutes to about 3 minutes, reached about 300 daily active users in limited release, and 78% of users reported the generated queries reduced the time they would have spent writing from scratch.

Reported metrics
Monthly interactive queries on data platformapproximately 1.2 million
Operations organization share of queries36%
manual SQL query authoring timearound 10 minutes
QueryGPT SQL generation timeabout 3 minutes
Show all 6 reported metrics
monthly interactive queries on data platformapproximately 1.2 million
Operations organization share of queries36%
manual SQL query authoring timearound 10 minutes
QueryGPT SQL generation timeabout 3 minutes
daily active users in limited releaseabout 300
users reporting reduced query writing time78%
Reported stack
large language modelsvector databasesRAGOpenAI GPT-4 Turbo
Source
https://www.uber.com/en-JP/blog/query-gpt/?uclick_id=eaf82e80-940f-4baf-87d6-76c4fbd37f1a
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

QueryGPT reduced SQL query authoring time from around 10 minutes to about 3 minutes, reached about 300 daily active users in limited release, and 78% of users reported the generated queries reduced the time they would…

What tools did this team use?

large language models, vector databases, RAG, OpenAI GPT-4 Turbo.

What results were reported?

Monthly interactive queries on data platform: approximately 1.2 million; Operations organization share of queries: 36%; manual SQL query authoring time: around 10 minutes; QueryGPT SQL generation time: about 3 minutes (source-reported, not independently verified).

What failed first in this deployment?

The initial version of QueryGPT used simple RAG over a small sample set and suffered declining accuracy as more tables were onboarded; simple similarity search between natural language prompts and SQL schemas returned…

How is this back office ops AI workflow structured?

User submits natural language prompt → Intent Agent classifies domain → Table Agent selects tables → User reviews and confirms tables → Column Prune Agent trims schemas → SQL query generated and returned → Evaluation and continuous improvement.

Related back office ops cases