Back office ops · Production

DoorDash builds a clusterless ML feature store serving 130M HMGETs per second

The problem

DoorDash's Redis-based ML feature store hit vertical scalability limits as the dataset grew, and a subsequent hybrid approach using a relational database added operational complexity without resolving the underlying scalability constraint.

First attempt

An initial Redis-only feature store hit per-instance vertical scale limits, and a hybrid Redis-plus-relational-database approach temporarily relieved the bottleneck but became unmanageable when the dataset doubled and the relational cluster reached 1,000-plus nodes.

Workflow diagram · grounded in source
1
Batch feature data upload to S3
integration
“All interesting feature data is consolidated from various sources via batch job and uploaded in parquet format to an S3 bucket.”
2
Per-shard indexing to RocksDB backup
integration
“A set of indexers — one per shard — processes this data, checkpointing the load progress into S3 in the form of a RocksDB backup.”
3
Serving layer fetches and serves shard
output
“The serving layer fetchers download a needs-based designated RocksDB shard backup and serve the data via RESP.”
4
RCM routes cluster topology
routing
“Redis Cluster Manager (RCM) implements the Redis cluster protocol to provide topology transparency to the clients. Based on the source IP address of the client, a subset of target shard instances are returned in the cluster nodes respons…”
5
Shadow validation double-read
validation
“the SPS client is configured to double-read features from both the existing native Redis cluster and our clusterless data store in parallel. Clients emitted key metrics describing such things as fetch latency, fetch value distribution, a…”
6
Gradual traffic migration
integration
“Using transport level routing controls, we began moving traffic exclusively to our feature store without the shadowing option. Occurring over a number of weeks, this migration built confidence while we continuously observed and addressed…”
Reported outcome

The clusterless ML feature store reached production, supporting a peak load of over 130M HMGETs per second for 1.6B retrieved features within a 50ms P999 latency target, with dynamically deployed capacity that grows with business demand.

Reported metrics
Sibyl Prediction Service peak ML evaluations per secondaround 900,000 ML evaluations per second
peak HMGETs per secondover 130M HMGETs per second
Features retrieved at peak load1.6B retrieved features
Fetch latency target50ms P999
Show all 7 reported metrics
Sibyl Prediction Service peak ML evaluations per secondaround 900,000 ML evaluations per second
peak HMGETs per secondover 130M HMGETs per second
features retrieved at peak load1.6B retrieved features
fetch latency target50ms P999
RAM vs SSD cost ratio100 times more expensive
relational database cluster size at scaling limit1,000-plus-node
feature store scalability and cost savings receptionso well received
Reported stack
Apache KvrocksRocksDBRedisS3Kubernetes
Source
https://careersatdoordash.com/blog/doordash-clusterless-ml-feature-store/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The clusterless ML feature store reached production, supporting a peak load of over 130M HMGETs per second for 1.6B retrieved features within a 50ms P999 latency target, with dynamically deployed capacity that grows w…

What tools did this team use?

Apache Kvrocks, RocksDB, Redis, S3, Kubernetes.

What results were reported?

Sibyl Prediction Service peak ML evaluations per second: around 900,000 ML evaluations per second; peak HMGETs per second: over 130M HMGETs per second; Features retrieved at peak load: 1.6B retrieved features; Fetch latency target: 50ms P999 (source-reported, not independently verified).

What failed first in this deployment?

An initial Redis-only feature store hit per-instance vertical scale limits, and a hybrid Redis-plus-relational-database approach temporarily relieved the bottleneck but became unmanageable when the dataset doubled and…

How is this back office ops AI workflow structured?

Batch feature data upload to S3 → Per-shard indexing to RocksDB backup → Serving layer fetches and serves shard → RCM routes cluster topology → Shadow validation double-read → Gradual traffic migration.

Related back office ops cases