Ecommerce ops · Production

DoorDash reduces ML model serving response time by 50% through gRPC optimization

The problem

DoorDash's gRPC-based ML model serving setup for search ranking had network overheads consuming up to 50% of service response time, creating large unexplained latency gaps at a scale of approximately one million predictions per second.

First attempt

A logging call placed outside the predictionsMade scope was creating approximately 50ms of overhead, identified only through transport-level gRPC tracing after the service was in production.

Workflow diagram · grounded in source
1
ML use cases trigger prediction requests
trigger
“All of the machine learning use cases at DoorDash such as search and recommendation, fraud detection, optimizing dasher dispatch, and predicting delivery times are served through Sibyl, our prediction service cluster”
2
Route requests to Sibyl cluster
routing
“The cluster is deployed as a Kubernetes service in AWS with each pod in the cluster composed of a Kotlin gRPC server”
3
Client-side load balancing
integration
“We use short-lived connections to the server that reset a client connection every 10 seconds to realize load-balancing”
4
Zstd payload compression
integration
“we found Zstandard (zstd) algorithm performs the best in terms of end-to-end service latency”
5
Model computation
ai_action
“our order recommendation use case runs a PyTorch model over embeddings of menu items, each of which have an excess of 100 dimensions”
6
Transport-level tracing to Wavefront
validation
“We use all the available hooks in gRPC tracing to get visibility into transport at client-side and server-side for our production traffic. These tracing measurements are then reported to Wavefront, our observability tool”
Reported outcome

By enabling client-side load balancing, switching to zstd payload compression, and removing a slow logging call identified via transport-level tracing, DoorDash reduced overall response time by 50% and network overheads by 33%.

Reported metrics
Overall response time reduction50%
Network overhead reduction33%
Network overhead share of response time (before)up to 50%
predictionsMade to outboundMessageSent gap reduction94%
Show all 9 reported metrics
overall response time reduction50%
network overhead reduction33%
network overhead share of response time (before)up to 50%
predictionsMade to outboundMessageSent gap reduction94%
latency improvement of zstd over gzipover 12%
search ranking predictions per second600k predictions/sec
overall prediction handling scale expectation1 to 10 million predictions/second
predictions per second currently handledapproximately one million predictions a second
time between predictionsMade and outboundMessageSent (before fix)~50ms
Reported stack
gRPCSibylKotlinKubernetesAWSPyTorchWavefrontZstandardNettyServerBuilderLinkerdIstiolzbench
Source
https://careersatdoordash.com/blog/enabling-efficient-machine-learning-model-serving/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By enabling client-side load balancing, switching to zstd payload compression, and removing a slow logging call identified via transport-level tracing, DoorDash reduced overall response time by 50% and network overhea…

What tools did this team use?

gRPC, Sibyl, Kotlin, Kubernetes, AWS, PyTorch, Wavefront, Zstandard, NettyServerBuilder, Linkerd.

What results were reported?

Overall response time reduction: 50%; Network overhead reduction: 33%; Network overhead share of response time (before): up to 50%; predictionsMade to outboundMessageSent gap reduction: 94% (source-reported, not independently verified).

What failed first in this deployment?

A logging call placed outside the predictionsMade scope was creating approximately 50ms of overhead, identified only through transport-level gRPC tracing after the service was in production.

How is this ecommerce ops AI workflow structured?

ML use cases trigger prediction requests → Route requests to Sibyl cluster → Client-side load balancing → Zstd payload compression → Model computation → Transport-level tracing to Wavefront.

Related ecommerce ops cases