Back office ops · Production

LyftLearn migrates ML training infrastructure from Kubernetes to AWS SageMaker hybrid architecture

The problem

Lyft's all-Kubernetes ML platform required custom orchestration logic for every new capability, suffered from unreliable state management via background watcher scripts, and consumed increasing engineering capacity on infrastructure rather than ML platform features.

First attempt

The fleet of background watcher scripts for synchronizing Kubernetes cluster state was inherently unreliable: training containers could succeed while Kubernetes marked jobs as failed, event streams timed out or arrived out of order, and container statuses transitioned inconsistently between watchers.

Workflow diagram · grounded in source

Job submission via multiple channels

trigger

“the LyftLearn UI for ad-hoc jobs, Airflow DAGs for scheduled training and batch prediction pipelines, and CI/CD pipelines that registered models along with their Docker images during deployments”

SageMaker Manager Service dispatch

integration

“LyftLearn Service interacts with a lean SageMaker Manager Service that only makes AWS SDK calls — it doesn't manage any low-level infrastructure”

Event-driven state management

integration

“We replaced the fleet of problematic background watchers with a single, reliable SQS consumer that processes status updates pushed from EventBridge”

ML training and batch execution

ai_action

“LyftLearn Compute runs on SageMaker, where the SageMaker Manager Service orchestrates training, batch processing, Hyperparameter Optimization (HPO), and JupyterLab notebooks through AWS SDK calls”

Model artifact storage and registry

output

“Training jobs in SageMaker generate model binaries and save them to S3. The Model Registry tracks these artifacts, and model serving services pull them for deployment”

Real-time model serving on Kubernetes

ai_action

“LyftLearn Serving runs on Kubernetes, powering a distributed architecture for real-time inference...handling production predictions for specific use cases (pricing, fraud, dispatch, ETA, etc.)”

Model health observability

feedback_loop

“LyftLearn Observability monitors model health and detects degradation across the platform. It tracks performance drift, identifies anomalies, scores model health, and monitors model activity to ensure production models maintain quality a…”

Reported outcome

Migrating LyftLearn Compute to SageMaker reduced infrastructure incidents significantly, cut notebook startup times by 40–50%, eliminated idle compute costs, and freed the ML platform team to focus on platform capabilities rather than low-level infrastructure management.

Reported metrics

Notebook startup time reduction40–50%

Infrastructure-related incidentsbecoming rare occurrences

ML training and batch compute costsreduced by eliminating idle cluster resources

Reported stack

KubernetesAWS SageMakerJupyterLabAirflowEventBridgeSQSS3ECREKSConfidantSparkSOCIEnvoyKubeflowKatibVizier

Source

https://eng.lyft.com/lyftlearn-evolution-rethinking-ml-platform-architecture-547de6c950e1

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Migrating LyftLearn Compute to SageMaker reduced infrastructure incidents significantly, cut notebook startup times by 40–50%, eliminated idle compute costs, and freed the ML platform team to focus on platform capabil…

What tools did this team use?

Kubernetes, AWS SageMaker, JupyterLab, Airflow, EventBridge, SQS, S3, ECR, EKS, Confidant.

What results were reported?

Notebook startup time reduction: 40–50%; Infrastructure-related incidents: becoming rare occurrences; ML training and batch compute costs: reduced by eliminating idle cluster resources (source-reported, not independently verified).

What failed first in this deployment?

How is this back office ops AI workflow structured?

Job submission via multiple channels → SageMaker Manager Service dispatch → Event-driven state management → ML training and batch execution → Model artifact storage and registry → Real-time model serving on Kubernetes → Model health observability.