Logistics ops · Production

DoorDash ML Platform builds a computational graph system and Python DSL for flexible ensemble model production serving

The problem

DoorDash's ML platform could not combine rule-based logic and models from multiple ML frameworks into a single production-ready ensemble because each framework has its own serialization format and C++ runtime library, and data scientists would need to write custom C++ production code for every model.

First attempt

When this work began, PyTorch had limited support for serializing computation graphs with Python dependencies. A proof of concept using TorchScript achieved similar performance but required significant setup effort due to bugs and incomplete documentation, making it impractical as a general solution.

Workflow diagram · grounded in source

Define ensemble model with Python DSL

trigger

“developing such an ensemble model only requires they: Define a static computation graph with Python DSL”

Serialize and upload to model store

integration

“Call helper function to upload to a model store, which serializes the graph in JSON and models in native format (text for LightGBM and TorchScript bytes for Pytorch)”

Sibyl loads serialized model

integration

“serialized model artifacts are loaded inside the Sibyl Prediction Service to serve real-time prediction requests with C++”

C++ graph evaluates nodes and sub-models

ai_action

“We use the xtensor matrix to store and perform computations for all the features as well as intermediate nodes. The computation nodes are iterated sequentially, performing computations and storing each nodes' value inside the matrix. If …”

Return prediction to client

output

“The final prediction result is passed back to Kotlin and returned to the client in real-time”

Reported outcome

The computational graph with C++ reduces CPU prediction time by more than a factor of 12 compared to Python and reduces total memory footprint from 120MB to 75MB.
The Python DSL reduces model definition from 800 lines of JSON to 20 lines of Python, enabling data scientists to build complex ensemble models serving a peak throughput of three million predictions per second.

Reported metrics

CPU prediction time reduction factormore than a factor of 12

Python 10,000 predictions time270s

C++ 10,000 predictions time20s

Python total memory footprint120MB

Show all 7 reported metrics

CPU prediction time reduction factormore than a factor of 12

Python 10,000 predictions time270s

C++ 10,000 predictions time20s

Python total memory footprint120MB

C++ total memory footprint75MB

Peak prediction throughputthree million predictions per second

Model definition lines reductionfrom 800 lines of JSON down to 20 lines of Python

Reported stack

LightGBMPyTorchSibyl Prediction ServicegRPCKubernetesKotlinxtensorJupyterGitDockerJNI

Source

https://careersatdoordash.com/blog/computational-graph-machine-learning-ensemble-model-support/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The computational graph with C++ reduces CPU prediction time by more than a factor of 12 compared to Python and reduces total memory footprint from 120MB to 75MB.

What tools did this team use?

LightGBM, PyTorch, Sibyl Prediction Service, gRPC, Kubernetes, Kotlin, xtensor, Jupyter, Git, Docker.

What results were reported?

CPU prediction time reduction factor: more than a factor of 12; Python 10,000 predictions time: 270s; C++ 10,000 predictions time: 20s; Python total memory footprint: 120MB (source-reported, not independently verified).

What failed first in this deployment?

When this work began, PyTorch had limited support for serializing computation graphs with Python dependencies.

How is this logistics ops AI workflow structured?

Define ensemble model with Python DSL → Serialize and upload to model store → Sibyl loads serialized model → C++ graph evaluates nodes and sub-models → Return prediction to client.