Netflix Auto Remediation uses ML to resolve 56% of Spark memory configuration errors without human intervention
Netflix's data platform runs hundreds of thousands of workflows and millions of jobs daily, but the rule-based error classifier could not automatically remediate memory configuration errors or handle the roughly half of job failures that went unclassified, requiring costly manual cross-team engineering effort for each incident.
The rule-based classifier Pensive could classify errors but not fix them: memory configuration errors still required manual expert remediation, and unclassified errors caused jobs to retry repeatedly with the default policy, incurring unnecessary compute costs.
Auto Remediation successfully remediates about 56% of all memory configuration errors without human intervention and reduces associated monetary costs by about 50% by applying correct configurations or disabling doomed retries.
Show all 5 reported metrics
Frequently asked questions
What did this team achieve with this AI workflow?
Auto Remediation successfully remediates about 56% of all memory configuration errors without human intervention and reduces associated monetary costs by about 50% by applying correct configurations or disabling doome…
What tools did this team use?
Pensive, Nightingale, ConfigService, Metaflow, Netflix Maestro, Ax library, MLP.
What results were reported?
Memory configuration errors auto-remediated: 56%; Monetary cost reduction from all errors: 50%; Memory configuration errors per month: 600; Job failures remaining unclassified: 50% (source-reported, not independently verified).
What failed first in this deployment?
The rule-based classifier Pensive could classify errors but not fix them: memory configuration errors still required manual expert remediation, and unclassified errors caused jobs to retry repeatedly with the default…
How is this incident management AI workflow structured?
Job failure triggers pipeline → Rule-based error routing → ML model predicts retry outcome → Bayesian optimizer recommends config → Recommendation stored in ConfigService → Scheduler applies new configuration → Daily model retraining.