Back office ops · Production

Netflix automates annualized A/B test impact projection using causal inference and machine learning

The problem

Netflix's Finance, Strategy & Analytics team manually forecasted signups, retention probabilities, and cumulative revenue for each A/B test on a one-year horizon — a process described as repetitive and time consuming.

Workflow diagram · grounded in source
1
A/B test allocation
trigger
“When we run an A/B test, we might allocate users for one month, and monitor results for only two billing periods.”
2
Short-term treatment effect observation
integration
“we have one member cohort, and we have two billing period treatment effects (𝜏.cohort1,period1 and 𝜏.cohort1,period2”
3
Surrogate index projection via Retention Model
ai_action
“We leverage our proprietary Retention Model and short-term observations — in the above example, 𝜏.1,2 — to estimate 𝜏.1,j , where j = 3…12.”
4
Cohort transportability assumption
validation
“we assume transportability: that each subsequent cohort's billing-period TE is the same as the first cohort's TE”
5
Annualized impact estimation
output
“We estimate the annualized impact by summing the values from each cohort.”
Reported outcome

Netflix built an automated approach empirically validated against long-running A/B tests and FS&A prior results, enabling quicker and more accurate estimates of the long-term value delivered by product features.

Reported metrics
Estimate speed and accuracy vs. prior manual approachquicker and more accurate estimates
Reported stack
Retention ModelDouble Machine Learning
Source
https://netflixtechblog.com/round-2-a-survey-of-causal-inference-applications-at-netflix-fd78328ee0bb
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Netflix built an automated approach empirically validated against long-running A/B tests and FS&A prior results, enabling quicker and more accurate estimates of the long-term value delivered by product features.

What tools did this team use?

Retention Model, Double Machine Learning.

What results were reported?

Estimate speed and accuracy vs. prior manual approach: quicker and more accurate estimates (source-reported, not independently verified).

How is this back office ops AI workflow structured?

A/B test allocation → Short-term treatment effect observation → Surrogate index projection via Retention Model → Cohort transportability assumption → Annualized impact estimation.

Related back office ops cases