Quality assurance · Production

Lyft builds a large-scale unsupervised ML model anomaly detection system with automated Slack alerting

The problem

ML model observability at Lyft was often neglected, and existing z-score-based anomaly detection generated too many false positives without per-scenario threshold tuning. Historically, domain-specific logic for each implementation made scaling a general-purpose solution across the organization impractical.

First attempt

The z-score based approach generated too many false positives unless thresholds were manually adjusted per scenario, making it impractical as a general-purpose first line of defense.

Workflow diagram · grounded in source
1
Data profiling with whylogs
integration
“We chose whylogs because we can build profiles of various functional, integral and distribution metrics with one pass. Another reason to choose whylogs for us is its low latency and the ability to fit into a MapReduce framework.”
2
Convert to time-series problem
ai_action
“After data profiling, the anomaly detection problem is converted to a standard time-series problem on the smaller profiles”
3
Forecast generation with StatsForecast
ai_action
“Statsforecast provides a wide range of statistical and econometric models to forecast univariate time series. With this package, we can easily choose from different models like ARIMA, MSTL, ETS and Exponential Smoothing that are neatly w…”
4
Anomaly isolation from confidence interval
ai_action
“An anomaly is any data that deviates from the confidence interval”
5
Feature drift root-cause analysis
ai_action
“we will train a regressor with changes in features over consecutive time periods and measure changes in predictions for the same time interval. We will then use Shapely values to explain the model”
6
Slack alert to model owner
output
“Upon detection of any anomaly, the corresponding Slack channel gets notified”
7
Mode Analytics dashboard
output
“We built a simple dashboard using Mode Analytics. Although these dashboards provide good insights into the model, the benefit is only realized with timely action on detected anomalies.”
Reported outcome

Models are automatically onboarded onto the detection system without user setup, drastically reducing turnaround time for acting on broken models.
Real-time detection catches anomalous predictions within a few minutes.

Reported metrics
Turnaround time to action on broken modelsdrastically reduced
Time to catch anomalous predictionswithin a few minutes
Reported stack
whylogsStatsForecastAutoARIMAFugueMode AnalyticsSlack
Source
https://eng.lyft.com/building-a-large-scale-unsupervised-model-anomaly-detection-system-part-2-3690f4c37c5b
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Models are automatically onboarded onto the detection system without user setup, drastically reducing turnaround time for acting on broken models.

What tools did this team use?

whylogs, StatsForecast, AutoARIMA, Fugue, Mode Analytics, Slack.

What results were reported?

Turnaround time to action on broken models: drastically reduced; Time to catch anomalous predictions: within a few minutes (source-reported, not independently verified).

What failed first in this deployment?

The z-score based approach generated too many false positives unless thresholds were manually adjusted per scenario, making it impractical as a general-purpose first line of defense.

How is this quality assurance AI workflow structured?

Data profiling with whylogs → Convert to time-series problem → Forecast generation with StatsForecast → Anomaly isolation from confidence interval → Feature drift root-cause analysis → Slack alert to model owner → Mode Analytics dashboard.

Related quality assurance cases