Logistics ops · Production

DoorDash retrains ML demand prediction models to restore accuracy after COVID-19 disruption

The problem

COVID-19 broke the historical demand patterns DoorDash's ML models relied on, making predictions inaccurate at a time when getting supply-demand balance right had direct consequences for both customers and Dasher earnings.

First attempt

The old Demand Prediction Model began severely underpredicting demand once COVID-19 restrictions took effect. During retraining, dataset manipulation hypotheses — including downsampling pre-pandemic data and removing holiday training data — proved negligible or detrimental to model performance.

Workflow diagram · grounded in source
1
COVID-19 breaks demand patterns
trigger
“The COVID-19 pandemic brought demand patterns higher and more volatile than ever before, making it necessary to retrain our prediction models to maintain performance”
2
Old model underpredicts demand
validation
“The old demand prediction model (DPM) began severely underpredicting demand once COVID-19 restrictions were put into place and an increasing number of consumers ordered food online”
3
Hyperparameter tuning
ai_action
“the number of estimators in our DPM was by far the most influential factor in its performance, allowing us to focus our hyperparameter tuning efforts”
4
Expanded training dataset
ai_action
“we expanded our training set to include all data from the past year, compared to the original PDM which was only trained on eight weeks of historical data. This expanded dataset significantly improved model performance”
5
Feature engineering via SHAP plots
ai_action
“we reformulated existing features into new representations that directly encode information relevant to our model. For example, we changed our prediction date feature from a timestamp to a categorical feature representing the day of the …”
6
Performance validation on test interval
validation
“we compared its performance within a three week test interval to the old DPM model. The old DPM achieved prediction variance of 7.9%, with its WDPP value falling within its percentile range 67% of the time. The new DPM yielded a predicti…”
7
Partial traffic deployment
output
“we partially deployed our new DPM by routing half of our prediction traffic to the new DPM and half to the old”
Reported outcome

The retrained DPM reduced prediction variance from 7.9% to 2.8% and increased the share of predictions within the SLA percentile range from 67% to 95%, yielding a model that was much more accurate while just as precise.

Reported metrics
new DPM prediction variance2.8%
old DPM prediction variance7.9%
new DPM percent predictions within SLA percentile range95%
old DPM percent predictions within SLA percentile range67%
Reported stack
Apache AirflowDatabricksPandasLightGBM
Source
https://careersatdoordash.com/blog/retraining-ml-models-covid-19/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The retrained DPM reduced prediction variance from 7.9% to 2.8% and increased the share of predictions within the SLA percentile range from 67% to 95%, yielding a model that was much more accurate while just as precise.

What tools did this team use?

Apache Airflow, Databricks, Pandas, LightGBM.

What results were reported?

new DPM prediction variance: 2.8%; old DPM prediction variance: 7.9%; new DPM percent predictions within SLA percentile range: 95%; old DPM percent predictions within SLA percentile range: 67% (source-reported, not independently verified).

What failed first in this deployment?

The old Demand Prediction Model began severely underpredicting demand once COVID-19 restrictions took effect.

How is this logistics ops AI workflow structured?

COVID-19 breaks demand patterns → Old model underpredicts demand → Hyperparameter tuning → Expanded training dataset → Feature engineering via SHAP plots → Performance validation on test interval → Partial traffic deployment.

Related logistics ops cases