Marketing ops · Production

Netflix builds a causal machine learning framework to identify successful visual components in promotional artwork

The problem

Netflix's vast and diverse catalog made scalable experimentation on promotional artwork impossible using traditional A/B tests, which investigated only one visual aspect at a time and required manual image labeling or creation of new asset variants, severely limiting how many titles could be tested simultaneously.

First attempt

Previous A/B tests investigated one artwork attribute at a time, typically within a single genre, which was not scalable due to manual labeling requirements and missed the multidimensional nature of artwork performance.

Workflow diagram · grounded in source
1
Artwork served and engagement logged
trigger
“Using Netflix's Artwork Personalization, we serve these assets to hundreds of millions of members everyday. To power this recommendation system, we look at user engagement patterns and see whether or not these engagements with artworks r…”
2
Computer vision feature extraction
ai_action
“we use a series of computer vision algorithms to gather objective image metadata, latent representation of the image, as well as some of the contextual metadata that a given image contains”
3
Association analysis and hypothesis formation
ai_action
“we utilize machine learning algorithms, consumer insights, and correlational analysis for discovering high-level associations between image features and an artwork's success. These statistically significant associations become our hypoth…”
4
Propensity and outcome model training
ai_action
“we have implemented the propensity model as a classifier (as we have a binary treatment variable — the presence of face) and the potential outcome model as a regressor (as we have a continuous outcome variable — adjusted take rate). We h…”
5
Causal forest ATE and CATE estimation
ai_action
“we have used a causal forest on the residuals of treatment and the outcome variables to capture the ATE, as well as CATE on different genres and countries”
6
Mediation and moderation analysis
ai_action
“In this dataset, we found that face size only partially mediates the effect of face count on asset effectiveness. This implies that both factors have an impact on asset effectiveness — fewer faces tend to be more effective even if we con…”
7
Creative insights delivered to team
output
“These insights will guide and assist our team of talented strategists and creatives to select and generate the most attractive artwork, leveraging the attributes that these models selected, down to a specific genre”
Reported outcome

Netflix developed a causal ML framework using Double ML and causal forests that reduces experimental effort to uncover causal relationships in artwork performance and delivers data-driven creative insights to strategists and creatives, enabling genre-level guidance on artwork selection.

Reported metrics
Face detection algorithm average precision~92% average precision
Reported stack
computer vision algorithmsXGBoostingcausal forestDouble ML
Source
https://netflixtechblog.com/causal-machine-learning-for-creative-insights-4b0ce22a8a96
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Netflix developed a causal ML framework using Double ML and causal forests that reduces experimental effort to uncover causal relationships in artwork performance and delivers data-driven creative insights to strategi…

What tools did this team use?

computer vision algorithms, XGBoosting, causal forest, Double ML.

What results were reported?

Face detection algorithm average precision: ~92% average precision (source-reported, not independently verified).

What failed first in this deployment?

Previous A/B tests investigated one artwork attribute at a time, typically within a single genre, which was not scalable due to manual labeling requirements and missed the multidimensional nature of artwork performance.

How is this marketing ops AI workflow structured?

Artwork served and engagement logged → Computer vision feature extraction → Association analysis and hypothesis formation → Propensity and outcome model training → Causal forest ATE and CATE estimation → Mediation and moderation analysis → Creative insights delivered to team.

Related marketing ops cases