Quality assurance · Production

Full-Spectrum ML Model Monitoring at Lyft

The problem

Lyft's ML models make millions of high-stakes decisions per day, but model performance degrades gradually and is hard to detect. Before a centralized solution, ML engineers built one-off monitoring per model, resulting in duplicated work and no centralized visibility across hundreds of models.

First attempt

When ETA models were retrained due to COVID-related demand drops, downstream pricing models that used ETAs as inputs dramatically under-predicted, revealing an unanticipated cascading dependency between models.

Workflow diagram · grounded in source
1
Model scores emitted to metrics
integration
“For every model scoring request made to our online model serving solution, LyftLearn Serving, the system emits the model output to our metrics system”
2
Online feature validation
validation
“This technique validates the features of every prediction request online against a set of expectations on that data”
3
Time-series score alerting
validation
“We can then define time-series based alerts on this data. Out of the box, the system checks for that models are not stuck emitting the same score over a period of time”
4
Scheduled anomaly detection
ai_action
“Anomaly detection is a technique that identifies potential model problems by analyzing statistical deviations of logged features and predictions over long periods of time. The calculation of aggregate metrics and the evaluation of deviat…”
5
Performance drift detection
validation
“The most popular use-case is to join model scores with their ground-truth data and calculate a performance metric on it”
6
Alert or dashboard output
output
“Model owners can be alerted about violations or check the performance of their production models on a dashboard”
Reported outcome

Over 90% of Lyft's production models have Feature Validation and Model Score Monitoring, and 75% have Performance Drift Detection or Anomaly Detection.
The system fired hundreds of alarms and caught over 15 high-impact issues in the nine months following general availability.

Reported metrics
production models with Feature Validation and Model Score Monitoringover 90%
production models with Performance Drift Detection or Anomaly Detection75%
High impact issues caught in first nine monthsover 15
Alarms firedhundreds of alarms
Show all 5 reported metrics
production models with Feature Validation and Model Score Monitoringover 90%
production models with Performance Drift Detection or Anomaly Detection75%
high impact issues caught in first nine monthsover 15
alarms firedhundreds of alarms
feature validation latency reductionover 500x to 0.1ms for a typical feature set
Reported stack
LyftLearn ServingGreat ExpectationsSparkFugueKubernetesVerity
Source
https://eng.lyft.com/full-spectrum-ml-model-monitoring-at-lyft-a4cdaf828e8f
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Over 90% of Lyft's production models have Feature Validation and Model Score Monitoring, and 75% have Performance Drift Detection or Anomaly Detection.

What tools did this team use?

LyftLearn Serving, Great Expectations, Spark, Fugue, Kubernetes, Verity.

What results were reported?

production models with Feature Validation and Model Score Monitoring: over 90%; production models with Performance Drift Detection or Anomaly Detection: 75%; High impact issues caught in first nine months: over 15; Alarms fired: hundreds of alarms (source-reported, not independently verified).

What failed first in this deployment?

When ETA models were retrained due to COVID-related demand drops, downstream pricing models that used ETAs as inputs dramatically under-predicted, revealing an unanticipated cascading dependency between models.

How is this quality assurance AI workflow structured?

Model scores emitted to metrics → Online feature validation → Time-series score alerting → Scheduled anomaly detection → Performance drift detection → Alert or dashboard output.

Related quality assurance cases