Lyft builds distributed ML model profiling pipeline for large-scale anomaly detection
LyftLearn hosts a large and growing number of ML models making hundreds of millions of predictions daily, with features and traffic patterns varying so widely across models that a single monitoring logic could not cover them all — and a prior z-score approach generated too many false positives to be useful.
A prior z-score-based anomaly detection approach produced too many false positives because model features and predictions can deviate statistically without implying a real problem, with seasonality being a key cause.
All model features and predictions are now automatically profiled daily via a distributed pipeline built on Spark, Fugue, and Whylogs, with new models onboarded automatically and no manual action required.
Frequently asked questions
What did this team achieve with this AI workflow?
All model features and predictions are now automatically profiled daily via a distributed pipeline built on Spark, Fugue, and Whylogs, with new models onboarded automatically and no manual action required.
What tools did this team use?
Spark, Whylogs, Fugue, Kubernetes, Hive.
What results were reported?
Model features and predictions profiling coverage: automatically profiled daily; Model onboarding effort: Users do not need to take any action when new models are introduced (source-reported, not independently verified).
What failed first in this deployment?
A prior z-score-based anomaly detection approach produced too many false positives because model features and predictions can deviate statistically without implying a real problem, with seasonality being a key cause.
How is this quality assurance AI workflow structured?
Inference request sampling → Distributed Whylogs profiling → Hourly-to-daily profile merge → Profile persistence.