Back office ops · Production

LyftLearn: ML model training and batch prediction infrastructure built on Kubernetes at Lyft

The problem

Lyft needed a unified platform to simplify ML model development, parallelize training, track runs, retrain on schedule, and deploy models across many teams using diverse modeling libraries and techniques.

Workflow diagram · grounded in source
1
Select hardware and base image
trigger
“the user needs to go to the LyftLearn homepage to select the hardware configuration (such as the number of GPU or CPU cores and memory) and a base image to start with”
2
Save model as container
output
“the user can Save Model, which saves a new container consisting of the model code and the additional dependencies overlaid on the base image. The user also needs to specify a version while saving”
3
Configure and schedule training jobs
integration
“The jobs can be configured and scheduled programmatically using the LyftLearn API or manually using CLI or GUI”
4
Run parallel training on Kubernetes
ai_action
“The training jobs run as Kubernetes jobs on the underlying Kubernetes cluster and can be scheduled to run periodically to retrain the model at a regular frequency. LyftLearn supports this parallelization via Flyte, Spark, or Fugue”
5
Deploy model for predictions
output
“the model could be deployed as a service and called by another online service for point predictions, such as a pricing model that is called for every ride request in the Lyft app, or (2) the model could be scheduled for periodic batch pr…”
6
Monitor via User Dashboard
feedback_loop
“Users can see all the models along with their corresponding versions and their past training and batch prediction runs on GUI. For each run, the user can access the corresponding logs and the model performance metrics”
Reported outcome

LyftLearn achieved wide adoption across dozens of teams building hundreds of models every week, with Kubernetes-based environment spin-up in seconds enabling the fast iteration critical to ML development.

Reported metrics
teams using LyftLearndozens of teams
Models built per weekhundreds of models every week
Environment spin-up timeonly a few seconds
Reported stack
KubernetesJupyterR-studioFlyteSparkFuguesklearnLightGBMXGBoostPyTorchTensorFlowAWS Elastic File SystemHivePrestoAWS RDS AuroraAWS Elastic Container RegistryDocker
Source
https://eng.lyft.com/lyftlearn-ml-model-training-infrastructure-built-on-kubernetes-aef8218842bb
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LyftLearn achieved wide adoption across dozens of teams building hundreds of models every week, with Kubernetes-based environment spin-up in seconds enabling the fast iteration critical to ML development.

What tools did this team use?

Kubernetes, Jupyter, R-studio, Flyte, Spark, Fugue, sklearn, LightGBM, XGBoost, PyTorch.

What results were reported?

teams using LyftLearn: dozens of teams; Models built per week: hundreds of models every week; Environment spin-up time: only a few seconds (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Select hardware and base image → Save model as container → Configure and schedule training jobs → Run parallel training on Kubernetes → Deploy model for predictions → Monitor via User Dashboard.

Related back office ops cases