Back office ops · Production

Uber's GenAI Gateway: unified LLM platform serving 16 million queries per month across close to 30 teams

The problem

Disparate LLM integration strategies across Uber's engineering teams led to inefficiencies and redundant efforts, while a rapidly growing number of LLM use cases made a centralized approach necessary.

Workflow diagram · grounded in source
1
Security review before access
human_review
“a standardized review process, managed by the Engineering Security team, reviews use cases against Uber's data handling standard before use cases are granted access to the gateway”
2
Developer submits LLM request
trigger
“developers write code as if they're using native OpenAI client, while being able to access LLMs from different vendors”
3
PII redaction
ai_action
“GenAI Gateway incorporates a PII redactor that anonymizes sensitive information within requests before forwarding them to third-party vendors”
4
LLM query via vendor or hosted model
ai_action
“offering seamless access to models from various vendors like OpenAI and Vertex AI, as well as Uber-hosted models, through a consistent and efficient interface”
5
PII un-redaction and response return
output
“Upon receiving responses from these external LLMs, the redacted entities are restored through an un-redaction process”
6
Audit logging and cost attribution
integration
“the generation of audit logs for comprehensive cost attribution, security audit purposes, quality evaluation, and so on”
Reported outcome

GenAI Gateway is used by close to 30 customer teams and serves 16 million queries per month with a peak QPS of 25, providing a single OpenAI-compatible interface to models from OpenAI, Vertex AI, and Uber-hosted LLMs.

Reported metrics
distinct LLM use cases at Uberover 60
customer teams using GenAI Gatewayclose to 30
Monthly queries served16 million
Peak queries per second25
Reported stack
GenAI GatewayOpenAIVertex AILangChainLlamaIndexSTOA
Source
https://www.uber.com/mx/en/blog/genai-gateway/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

GenAI Gateway is used by close to 30 customer teams and serves 16 million queries per month with a peak QPS of 25, providing a single OpenAI-compatible interface to models from OpenAI, Vertex AI, and Uber-hosted LLMs.

What tools did this team use?

GenAI Gateway, OpenAI, Vertex AI, LangChain, LlamaIndex, STOA.

What results were reported?

distinct LLM use cases at Uber: over 60; customer teams using GenAI Gateway: close to 30; Monthly queries served: 16 million; Peak queries per second: 25 (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Security review before access → Developer submits LLM request → PII redaction → LLM query via vendor or hosted model → PII un-redaction and response return → Audit logging and cost attribution.

Related back office ops cases