Finance ops · Production

Netflix builds ML observability framework to bring transparency to payment routing decisions

The problem

Netflix routes thousands of payment transactions per minute with ML models, but as the ML portfolio grew, ad-hoc monitoring became insufficient — stakeholders lacked visibility into why models made specific routing decisions and could not detect or diagnose issues at scale.

Workflow diagram · grounded in source
1
Payment transaction arrives
trigger
“routing thousands of payment transactions per minute”
2
ML model scores and routes
ai_action
“passes these to a model which creates some score between 0 and 1, and then that score is translated into a decision (say, whether to process a card as Debit or Credit)”
3
Penalty layers adjust routing
routing
“different penalty or guardrail layers impact final volume as you move left to right. For example, the model originally allocated 22% traffic to processor W with Configuration A, however for cost and contractual considerations, the traffi…”
4
Observability logging
integration
“the raw data, the final features that fed the model, a unique identifier for the model, the feature importances for that model, the raw model score, the cutoffs used to map a score to a decision, timestamps for the decision as well as th…”
5
Monitoring detects issues
validation
“a good observability system would detect aberrations in input data, the feature pipeline, predictions, and outcomes as well provide insight into the likely causes of model decisions and/or performance”
6
SHAP explains routing decisions
ai_action
“we leverage SHAP as one core algorithm to unpack a variety of models and open the black box for stakeholders”
7
Stakeholder insights delivered
output
“The explanation system not only demystifies our machine learning models but also fosters transparency and trust among our stakeholders, enabling more informed and confident decision-making”
Reported outcome

The observability framework delivered a massive operational complexity reduction and improved transaction approval rate, while giving stakeholders the transparency needed to trust and act on ML routing decisions.

Reported metrics
Operational complexitymassive operational complexity reduction
Transaction approval rateimproved transaction approval rate
Reported stack
SHAP
Source
https://netflixtechblog.com/ml-observability-bring-transparency-to-payments-and-beyond-33073e260a38
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The observability framework delivered a massive operational complexity reduction and improved transaction approval rate, while giving stakeholders the transparency needed to trust and act on ML routing decisions.

What tools did this team use?

SHAP.

What results were reported?

Operational complexity: massive operational complexity reduction; Transaction approval rate: improved transaction approval rate (source-reported, not independently verified).

How is this finance ops AI workflow structured?

Payment transaction arrives → ML model scores and routes → Penalty layers adjust routing → Observability logging → Monitoring detects issues → SHAP explains routing decisions → Stakeholder insights delivered.

Related finance ops cases