Back office ops · Production

Uber QueryGPT: Natural Language to SQL Generation Using Generative AI

The problem

SQL query authoring at Uber requires both SQL expertise and deep knowledge of internal data models, consuming around 10 minutes per query across approximately 1.2 million monthly queries — a significant productivity bottleneck, especially for the Operations organization which accounts for about 36% of queries.

First attempt

The initial RAG-based prototype of QueryGPT saw declining accuracy as more tables were onboarded; naive similarity search on schema samples failed to return relevant results for natural language prompts, and large schemas with 200+ columns exceeded the 32K token limit of the available model, breaking query generation.

Workflow diagram · grounded in source
1
User submits natural language question
trigger
“enabling users to generate SQL queries through natural language prompts”
2
Intent Agent classifies domain
ai_action
“Every incoming prompt from the user now first runs through an "intent" agent. The purpose of this intent agent is to map the user's question to one or more business domains/workspaces”
3
Table Agent selects tables
ai_action
“we added another LLM agent (Table Agent) that would pick the right tables and send those out to the user to either "ACK" or edit the given list and set the right tables”
4
User reviews and approves tables
human_review
“The user would either select the "Looks Good" button or edit the existing list and modify the list of tables to be used by the LLM for query generation”
5
Column Prune Agent trims schemas
ai_action
“we implemented a "Column Prune" agent, wherein we use an LLM call to prune the irrelevant columns from the schemas we provided to the LLM”
6
LLM generates SQL query
ai_action
“QueryGPT uses large language models (LLM), vector databases, and similarity search to generate complex queries from English questions that are provided by the user as input”
7
Output: SQL query and explanation
output
“The response would include a SQL Query and an explanation from the LLM about how the query was generated”
8
Evaluation feedback loop
feedback_loop
“To track incremental improvements in QueryGPT's performance, we needed a standardized evaluation procedure. This enabled us to differentiate between repeated vs. anomalous shortcomings of the service”
Reported outcome

QueryGPT reduces query authoring from about 10 minutes to about 3 minutes; in its limited release to Operations and Support teams it averages 300 daily active users, with about 78% reporting that generated queries reduced the time they would have spent writing from scratch.

Reported metrics
query authoring time before QueryGPTabout 10 minutes
query authoring time with QueryGPTabout 3 minutes
monthly interactive queries at Uberapproximately 1.2 million
Operations organization share of queriesabout 36%
Show all 6 reported metrics
query authoring time before QueryGPTabout 10 minutes
query authoring time with QueryGPTabout 3 minutes
monthly interactive queries at Uberapproximately 1.2 million
Operations organization share of queriesabout 36%
daily active users (limited release)about 300
users reporting time savingsabout 78%
Reported stack
LLMvector databasessimilarity searchRAGOpenAI GPT-4 Turbok-nearest neighbor search
Source
https://www.uber.com/en-IN/blog/query-gpt/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

QueryGPT reduces query authoring from about 10 minutes to about 3 minutes; in its limited release to Operations and Support teams it averages 300 daily active users, with about 78% reporting that generated queries red…

What tools did this team use?

LLM, vector databases, similarity search, RAG, OpenAI GPT-4 Turbo, k-nearest neighbor search.

What results were reported?

query authoring time before QueryGPT: about 10 minutes; query authoring time with QueryGPT: about 3 minutes; monthly interactive queries at Uber: approximately 1.2 million; Operations organization share of queries: about 36% (source-reported, not independently verified).

What failed first in this deployment?

The initial RAG-based prototype of QueryGPT saw declining accuracy as more tables were onboarded; naive similarity search on schema samples failed to return relevant results for natural language prompts, and large sch…

How is this back office ops AI workflow structured?

User submits natural language question → Intent Agent classifies domain → Table Agent selects tables → User reviews and approves tables → Column Prune Agent trims schemas → LLM generates SQL query → Output: SQL query and explanation → Evaluation feedback loop.

Related back office ops cases