Compliance monitoring · Production

DoorDash dark shipping: safely iterating ML fraud detection models in production

The problem

DoorDash's anti-fraud ML models must meet stringent reliability and correctness requirements while needing rapid iteration as fraud patterns evolve — a tension that pre-production testing alone cannot resolve because correctness can only be fully verified with real production traffic.

Workflow diagram · grounded in source
1
Pre-production iteration
validation
“Before a model goes to production, it is iterated rapidly and extensively in the development environments, where it is updated, trained, evaluated, and tuned, with a turnaround time ranging from minutes to hours. Once the backtesting res…”
2
Shadow traffic at 1% volume
validation
“At this time, the model is exercised safely (at low volume and with shadow traffic only), while in a true production environment, end-to-end. This allows us to verify multiple things: There are no errors due to misconfiguration, missing …”
3
Shadow traffic at 100% volume
validation
“We can now ramp up the shadow traffic to 100% of the volume, which serves two purposes: We can analyze model performance without risking any adverse business impact. We can make sure there's no undue deterioration of system metrics due t…”
4
A/B experiment: incumbent vs. new model
validation
“we use the DoorDash Curie experimentation system, setting up an experiment that compares the performance of the old and the new models in a rigorous evaluation”
5
Full live traffic ramp
output
“Once we see statistically significant improvement, the new model is ramped up to receive 100% of the live traffic”
Reported outcome

The dark rollout practice enables DoorDash to iterate on production ML deployments quickly while minimizing the risk of regressions, by validating models with real production traffic before they make live decisions.

Reported metrics
ML model daily invocation volumemany millions of times each day
Deployment speed and regression risk outcomeiterate on production ML deployments quickly while minimizing risk of regressions
Reported stack
machine learning platformrule engineML serviceCurie experimentation system
Source
https://careersatdoordash.com/blog/ship-to-production-darkly-moving-fast-staying-safe-with-ml-deployments/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The dark rollout practice enables DoorDash to iterate on production ML deployments quickly while minimizing the risk of regressions, by validating models with real production traffic before they make live decisions.

What tools did this team use?

machine learning platform, rule engine, ML service, Curie experimentation system.

What results were reported?

ML model daily invocation volume: many millions of times each day; Deployment speed and regression risk outcome: iterate on production ML deployments quickly while minimizing risk of regressions (source-reported, not independently verified).

How is this compliance monitoring AI workflow structured?

Pre-production iteration → Shadow traffic at 1% volume → Shadow traffic at 100% volume → A/B experiment: incumbent vs. new model → Full live traffic ramp.

Related compliance monitoring cases