Back office ops · Production

DoorDash builds out-of-the-box ML model observability platform to detect and prevent model drift

The problem

DoorDash's ML models degraded over time after deployment — a process called model drift — negatively impacting accuracy of time estimates and other model outputs. The team had no systematic monitoring capability, so when models made incorrect predictions, diagnosing the cause took a long time and forced the engineering team to spend significant effort on reactive investigation.

First attempt

Storing prediction logs in a data warehouse supported ad-hoc deep dives but provided no big-picture visibility into why models were drifting, leaving the team without a proactive way to detect or diagnose drift.

Workflow diagram · grounded in source
1
Sibyl logs predictions
integration
“The prediction logs consist of every prediction made by a model, including the prediction result, prediction ID, feature values, and object identifiers that were used to make that prediction”
2
Hourly SQL aggregation
integration
“Both hourly and daily we plug in the duration, the predictor name, and the model ID into this SQL query template, generate the final SQL, query the data warehouse”
3
Prometheus metric emission
output
“once we receive the aggregated value, emit this information as a Prometheus metric”
4
Grafana dashboard visualization
output
“we can view the trends of feature value statistics using the Grafana dashboard, which includes graphs for a specific predictor name, model ID and feature name”
5
Threshold alerting
output
“create queries using PromQL, add thresholds, and connect alerts to either a team-specific Slack channel or a team-specific PagerDuty”
6
Data scientist review
human_review
“Our customers are able to view the trends in charts, zoom in and out of time ranges, and compare trends across different periods of time”
Reported outcome

DoorDash shipped a scalable, out-of-the-box ML monitoring platform that onboarded multiple teams including Logistics, Fraud, Supply and Demand, and ETA, enabling self-serve alerting and freeing data scientists to focus on model development rather than systems design.

Reported metrics
Data scientist focusdata scientists focus on model development rather than systems design
Teams onboardedmany teams including Logistics, Fraud, Supply and Demand, and ETA teams
Reported stack
SibylPrometheusGrafanaPromQLApache SparkYAMLSlackPagerDuty
Source
https://careersatdoordash.com/blog/monitor-machine-learning-model-drift/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

DoorDash shipped a scalable, out-of-the-box ML monitoring platform that onboarded multiple teams including Logistics, Fraud, Supply and Demand, and ETA, enabling self-serve alerting and freeing data scientists to focu…

What tools did this team use?

Sibyl, Prometheus, Grafana, PromQL, Apache Spark, YAML, Slack, PagerDuty.

What results were reported?

Data scientist focus: data scientists focus on model development rather than systems design; Teams onboarded: many teams including Logistics, Fraud, Supply and Demand, and ETA teams (source-reported, not independently verified).

What failed first in this deployment?

Storing prediction logs in a data warehouse supported ad-hoc deep dives but provided no big-picture visibility into why models were drifting, leaving the team without a proactive way to detect or diagnose drift.

How is this back office ops AI workflow structured?

Sibyl logs predictions → Hourly SQL aggregation → Prometheus metric emission → Grafana dashboard visualization → Threshold alerting → Data scientist review.

Related back office ops cases