Back office ops · Production

Lyft's ML Feature Serving Infrastructure: Unified Batch and Streaming Feature Access for Training and Online Inference

The problem

Lyft's ML models require features computed via both batch jobs on the data warehouse and real-time event streams, and those features must be accessible in two modes: batch queries for model training and low-latency point lookups for online inference — a dual-access requirement that needed a unified infrastructure solution.

Workflow diagram · grounded in source
1
SQL Feature Definition
trigger
“The SQL is expected to have one column designated to be an entity ID (which is an identifier for the business entity such as a driver, passenger, or ride), and the rest of the columns are features”
2
Batch Ingestion via Flyte
integration
“For features defined on batch data, we run scheduled feature extraction jobs using Flyte”
3
Stream Ingestion via Flink
integration
“For features defined on streaming data, the ingestion job happens as a custom Flink job”
4
Validation and DynamoDB Storage
validation
“it validates the feature value against the feature metadata and then writes the feature value in DynamoDB”
5
Replication and Redis Caching
integration
“From DynamoDB, the feature values are replicated to Hive and Elasticsearch. We use Redis as a write-through cache for both feature values and feature metadata to increase read throughput”
6
Online Inference Retrieval
output
“reads first from a Redis cache, and then from DynamoDB in case of a cache-miss”
7
Training Data Access via Hive
output
“the replicated data to Hive is used. The features are stored as tables in Hive, which are then queried by the training process directly”
Reported outcome

The Feature Service has been widely adopted across Lyft teams since Q4 2017, hosting thousands of features across many ML models, serving millions of requests per minute with single-digit millisecond latency and 99.99%+ availability.

Reported metrics
Requests served per minutemillions of requests per minute
Read latencysingle-digit millisecond latency
Service availability99.99%+
Features hostedseveral 1000s of features
Reported stack
FlyteFlinkDynamoDBHiveRedisElasticsearchKafka
Source
https://eng.lyft.com/ml-feature-serving-infrastructure-at-lyft-d30bf2d3c32a
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The Feature Service has been widely adopted across Lyft teams since Q4 2017, hosting thousands of features across many ML models, serving millions of requests per minute with single-digit millisecond latency and 99.99…

What tools did this team use?

Flyte, Flink, DynamoDB, Hive, Redis, Elasticsearch, Kafka.

What results were reported?

Requests served per minute: millions of requests per minute; Read latency: single-digit millisecond latency; Service availability: 99.99%+; Features hosted: several 1000s of features (source-reported, not independently verified).

How is this back office ops AI workflow structured?

SQL Feature Definition → Batch Ingestion via Flyte → Stream Ingestion via Flink → Validation and DynamoDB Storage → Replication and Redis Caching → Online Inference Retrieval → Training Data Access via Hive.

Related back office ops cases