Back office ops · Production

Building real-time machine learning foundations at Lyft with RealtimeMLPipeline

The problem

Streaming data was not a first-class citizen in Lyft's LyftLearn ML platform, forcing teams to spend weeks or months of engineering effort to integrate it into ML workflows despite strong developer appetite for real-time ML systems.

Workflow diagram · grounded in source
1
Developer defines pipeline
trigger
“a developer provides metadata such as a feature name and version, a query to compute it, and instantiates a RealtimeMLPipeline Python object”
2
Notebook testing on streaming data
validation
“test it on realistic streaming data (in that same notebook) to ensure their code is functionally and logically correct”
3
Commit and deploy to production
integration
“commit the code to a GitHub repository, and deploy the pipeline to production”
4
Event-driven ML computation
ai_action
“Making decisions, e.g. retraining a model, triggering an alert, or running an inference call, with real-time streaming data”
5
Features delivered to Feature Storage
output
“outputs computed features to Kafka which in turn delivers them to our Feature Storage infrastructure”
Reported outcome

Lyft reduced the time to launch a new real-time ML application from multiple weeks to a few days, achieved self-service adoption across nearly all engineering pillars (Rider, Driver, Marketplace, Mapping, Safety), and enabled teams to build higher-order abstractions including a Real-time Anomaly Detection product.

Reported metrics
time to build RealtimeMLPipelinefrom many weeks to days
time to launch new real-time ML application todaya few days
Prior integration effort per use caseweeks or months of engineering effort
Reported stack
FlinkPyFlinkKafkaKinesisS3KubernetesGitHubJupyterLyftLearnHive
Source
https://eng.lyft.com/building-real-time-machine-learning-foundations-at-lyft-6dd99b385a4e
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Lyft reduced the time to launch a new real-time ML application from multiple weeks to a few days, achieved self-service adoption across nearly all engineering pillars (Rider, Driver, Marketplace, Mapping, Safety), and…

What tools did this team use?

Flink, PyFlink, Kafka, Kinesis, S3, Kubernetes, GitHub, Jupyter, LyftLearn, Hive.

What results were reported?

time to build RealtimeMLPipeline: from many weeks to days; time to launch new real-time ML application today: a few days; Prior integration effort per use case: weeks or months of engineering effort (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Developer defines pipeline → Notebook testing on streaming data → Commit and deploy to production → Event-driven ML computation → Features delivered to Feature Storage.

Related back office ops cases