Incident management · Production

Netflix Auto Remediation uses ML to resolve 56% of Spark memory configuration errors without human intervention

The problem

Netflix's data platform runs hundreds of thousands of workflows and millions of jobs daily, but the rule-based error classifier could not automatically remediate memory configuration errors or handle the roughly half of job failures that went unclassified, requiring costly manual cross-team engineering effort for each incident.

First attempt

The rule-based classifier Pensive could classify errors but not fix them: memory configuration errors still required manual expert remediation, and unclassified errors caused jobs to retry repeatedly with the default policy, incurring unnecessary compute costs.

Workflow diagram · grounded in source
1
Job failure triggers pipeline
trigger
“Upon a job failure, Scheduler calls Pensive to get the error classification”
2
Rule-based error routing
routing
“Pensive classifies the error based on the rule-based classifier. If the error is identified to be a memory configuration error or an unclassified error, it calls Nightingale to get recommendations”
3
ML model predicts retry outcome
ai_action
“A prediction model that jointly estimates a) probability of retry success, and b) retry cost in dollars, conditional on properties of the retry”
4
Bayesian optimizer recommends config
ai_action
“We use Bayesian Optimization (implemented via Meta's Ax library) to explore the configuration space and generate a recommendation”
5
Recommendation stored in ConfigService
integration
“Pensive updates the error classification result and saves the recommended configurations to ConfigService”
6
Scheduler applies new configuration
output
“Scheduler can then restart the job with the mutated configurations (including the recommended configurations)”
7
Daily model retraining
feedback_loop
“The prediction model is retrained offline daily”
Reported outcome

Auto Remediation successfully remediates about 56% of all memory configuration errors without human intervention and reduces associated monetary costs by about 50% by applying correct configurations or disabling doomed retries.

Reported metrics
Memory configuration errors auto-remediated56%
Monetary cost reduction from all errors50%
Memory configuration errors per month600
Job failures remaining unclassified50%
Show all 5 reported metrics
memory configuration errors auto-remediated56%
monetary cost reduction from all errors50%
memory configuration errors per month600
job failures remaining unclassified50%
rules added to classifier300+
Reported stack
PensiveNightingaleConfigServiceMetaflowNetflix MaestroAx libraryMLP
Source
https://netflixtechblog.com/evolving-from-rule-based-classifier-machine-learning-powered-auto-remediation-in-netflix-data-039d5efd115b
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Auto Remediation successfully remediates about 56% of all memory configuration errors without human intervention and reduces associated monetary costs by about 50% by applying correct configurations or disabling doome…

What tools did this team use?

Pensive, Nightingale, ConfigService, Metaflow, Netflix Maestro, Ax library, MLP.

What results were reported?

Memory configuration errors auto-remediated: 56%; Monetary cost reduction from all errors: 50%; Memory configuration errors per month: 600; Job failures remaining unclassified: 50% (source-reported, not independently verified).

What failed first in this deployment?

The rule-based classifier Pensive could classify errors but not fix them: memory configuration errors still required manual expert remediation, and unclassified errors caused jobs to retry repeatedly with the default…

How is this incident management AI workflow structured?

Job failure triggers pipeline → Rule-based error routing → ML model predicts retry outcome → Bayesian optimizer recommends config → Recommendation stored in ConfigService → Scheduler applies new configuration → Daily model retraining.