Customer support · Production

How Uber's Michelangelo ML platform evolved from predictive models to generative AI at scale

The problem

Before Michelangelo, Uber's ML development was fragmented: applied scientists used Jupyter Notebooks and engineers built bespoke deployment pipelines with no system for reliable reproducible workflows, no easy way to compare training experiments, and no established path to production deployment.

First attempt

Michelangelo 1.0 had four structural gaps: no comprehensive ML quality definition or project tiering, insufficient deep learning support, inadequate collaborative model development capabilities, and fragmented ML tooling forcing developers to constantly switch between semi-isolated systems.

Workflow diagram · grounded in source
1
Unified ML project initiation
trigger
“MA Studio provides a simplified user flow covering every step of the ML journey from feature/data prep, model training, deployment, all the way to production performance monitoring, and model CI/CD, all in one place”
2
Feature engineering and storage
integration
“a feature store named Palette was built to better manage and share feature pipelines across teams. It supported both batch and near-real-time feature computation use cases. Currently, Palette hosts more than 20,000 features that can be l…”
3
Distributed model training
ai_action
“Michelangelo 2.0 supports both TensorFlow and PyTorch frameworks for large-scale DL model training by leveraging our distributed training framework Horovod”
4
Model quality evaluation (MES)
validation
“we launched Model Excellence Score (MES), a framework for measuring and monitoring key dimensions and metrics at each stage of a model's life cycle, such as training model accuracy, prediction accuracy, model freshness, and prediction fe…”
5
Safe incremental deployment
output
“Safe and incremental zonal rollout, automatic rollback triggers, and production runtime validation”
6
Production monitoring and observability
feedback_loop
“Built-in and unified ML observability toolkit: Model performance monitoring, feature monitoring, online/offline feature consistency check, and MES”
Reported outcome

Michelangelo now manages approximately 400 active ML projects with over 20K model training jobs monthly, more than 5K models in production serving 10 million real-time predictions per second at peak, and deep learning adoption in tier-1 projects increased from almost zero to more than 60%.

Reported metrics
active ML projects on Michelangeloapproximately 400
Model training jobs per monthover 20K
Models in productionmore than 5K
Real-time predictions per second at peak10 million
Show all 12 reported metrics
active ML projects on Michelangeloapproximately 400
model training jobs per monthover 20K
models in productionmore than 5K
real-time predictions per second at peak10 million
daily trips served on platform25 million
monthly active users137 million
tier-1 models using deep learningmore than 60%
DL tier-1 adoption growthincreased from almost zero to more than 60%
Palette features availablemore than 20,000
GPUs managed by Ubermore than five thousand
cities of operationover 10,000
countries of operationmore than 70
Reported stack
MichelangeloPaletteHorovodRayTensorFlowPyTorchXGBoostSparkTritonKubernetesCanvasPyMLGalleryManifoldNeuropodDockerBazelJupyter NotebooksetcdPelotonMesosGCPOCI
Source
https://www.uber.com/us/en/blog/from-predictive-to-generative-ai/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Michelangelo now manages approximately 400 active ML projects with over 20K model training jobs monthly, more than 5K models in production serving 10 million real-time predictions per second at peak, and deep learning…

What tools did this team use?

Michelangelo, Palette, Horovod, Ray, TensorFlow, PyTorch, XGBoost, Spark, Triton, Kubernetes.

What results were reported?

active ML projects on Michelangelo: approximately 400; Model training jobs per month: over 20K; Models in production: more than 5K; Real-time predictions per second at peak: 10 million (source-reported, not independently verified).

What failed first in this deployment?

Michelangelo 1.0 had four structural gaps: no comprehensive ML quality definition or project tiering, insufficient deep learning support, inadequate collaborative model development capabilities, and fragmented ML tool…

How is this customer support AI workflow structured?

Unified ML project initiation → Feature engineering and storage → Distributed model training → Model quality evaluation (MES) → Safe incremental deployment → Production monitoring and observability.

Related customer support cases