Logistics ops · Production

DoorDash best practices for regression-free ML model migration in Dasher assignment

The problem

Migrating ML models from DoorDash's legacy monolith to a microservices architecture posed high regression risk because model complexity—with dozens or hundreds of interacting features—made regressions hard to prevent, detect, and correct, and different state snapshots in the new service altered model inputs and outputs.

First attempt

During the sequential swap of inference sources, two swaps produced statistically significant degradations in ASAP and DAT metrics, with degradations worse than the secondary success criterion, requiring corrective measures before continuing.

Workflow diagram · grounded in source
1
Define metrics and success thresholds
validation
“two are particularly important. One, called ASAP, measures how long it takes consumers to get their orders, and the other, called DAT, focuses on how long a Dasher takes to deliver the order”
2
Identify and isolate risky components
routing
“migrating the models to the new server and migrating the client call to a new client service each imposed a migration regression risk. We therefore wanted to carry them out separately.”
3
Migrate models to Sibyl server
integration
“The models would be served on Sibyl, DoorDash's newest generation online prediction service. In particular, our models are built with Python for fast development, but Sibyl stores the serialized version of the models and evaluates them w…”
4
Verify server inference consistency
validation
“we are able to verify the consistency by simply comparing every pair of inferences made by the two servers. As long as the inferences were the same, the assignment decisions would be the same”
5
Sequential inference source swap
integration
“we only swapped the "source" of inferences of one model at a time. That is, for one model at a time, the assignment client changed to use the inferences made by itself from ones given by the legacy monolith client.”
6
Switchback experiment per swap
validation
“we ran a switchback experiment for each swap: the control group used the inferences from the model of interest requested by the old monolith client and the treatment group used ones done by the assignment client”
7
Regression detection and correction
human_review
“because each swap only involved one model and its related features, we were able to quickly find the root cause and design corrective measures”
8
Re-validate and complete migration
output
“we re-ran the switchback experiment for the same swap after corrective measures had been applied. Everything turned out to be okay, and we moved to the next swap. Once all switchback experiment tests passed, all the inferences used by th…”
Reported outcome

After corrective measures were applied and switchback experiments re-run for each affected swap, all tests passed and the migration completed successfully, maintaining Dasher assignment quality.

Reported metrics
Migration outcomereached our business objective and metrics fairly quickly
Performance degradation events detected and correctedtwo swaps had led to performance degradations
Reported stack
SibylPythonC++
Source
https://careersatdoordash.com/blog/best-practices-for-regression-free-machine-learning-model-migrations/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After corrective measures were applied and switchback experiments re-run for each affected swap, all tests passed and the migration completed successfully, maintaining Dasher assignment quality.

What tools did this team use?

Sibyl, Python, C++.

What results were reported?

Migration outcome: reached our business objective and metrics fairly quickly; Performance degradation events detected and corrected: two swaps had led to performance degradations (source-reported, not independently verified).

What failed first in this deployment?

During the sequential swap of inference sources, two swaps produced statistically significant degradations in ASAP and DAT metrics, with degradations worse than the secondary success criterion, requiring corrective me…

How is this logistics ops AI workflow structured?

Define metrics and success thresholds → Identify and isolate risky components → Migrate models to Sibyl server → Verify server inference consistency → Sequential inference source swap → Switchback experiment per swap → Regression detection and correction → Re-validate and complete migration.

Related logistics ops cases