Flyte: Lyft's ML orchestration platform powers 1M+ pipelines and joins LF AI & Data
Lyft needed to orchestrate complex ML workflows for its ETA product, requiring management of large historical training datasets, complex output artifacts, backtesting, frequent retraining, and simultaneous multi-model deployment — with the largest bottleneck being infrastructure procurement and management for models that might not work out.
The initial v1 of Flyte used AWS Step Functions as its scheduler, which proved too rigid to extend with new features natively, leading the team to build a container-native scheduling engine.
By mid 2020 Flyte was powering more than 1 million pipelines at Lyft across ETA, Pricing, Mapping, Driver Engagement, Growth, and Map generation teams, and was contributed to the Linux Foundation AI & Data as its 25th hosted project.
Frequently asked questions
What did this team achieve with this AI workflow?
By mid 2020 Flyte was powering more than 1 million pipelines at Lyft across ETA, Pricing, Mapping, Driver Engagement, Growth, and Map generation teams, and was contributed to the Linux Foundation AI & Data as its 25th…
What tools did this team use?
Flyte, AWS Step Functions, flytekit, AWS, GCP.
What results were reported?
pipelines powered at Lyft: more than 1 million; Platform adoption growth: adoption grew exponentially; total cost of running Flyte: total cost of running Flyte at Lyft was low (source-reported, not independently verified).
What failed first in this deployment?
The initial v1 of Flyte used AWS Step Functions as its scheduler, which proved too rigid to extend with new features natively, leading the team to build a container-native scheduling engine.
How is this back office ops AI workflow structured?
Historical data training trigger → Multi-model development and deployment → Backtesting and hypothesis validation → Artifact output generation → Production retraining loop.