Back office ops · Production

Netflix builds a human-augmenting agentic workflow for observational causal inference

The problem

Observational causal inference (OCI) requires substantial judgment and domain expertise, but repetitive aspects like rechecking covariate balance, conducting sensitivity analyses, and tracking multiple iterations are error-prone — and LLMs given unscaffolded analysis plans produce biased estimates, as demonstrated by early adopter bias inflating the Netflix case study baseline.

First attempt

One-shot LLM prompting without scaffolding produced consistently wrong answers on benchmark datasets; in the Netflix case study the paved-path agentic workflow produced an updated estimate that was just 25% of the baseline, revealing that the unscaffolded approach was heavily distorted by early adopter bias and poor overlap.

Workflow diagram · grounded in source
1
Principal submits analysis plan
trigger
“Principals: Provide an initial analysis plan containing its context and goals. Provide context on the main threats to valid inference and the confounders that must be controlled. Specify the tools that can be used for the analysis. Speci…”
2
Actor refines plan and executes analysis
ai_action
“Actors: Refine the principal's plan into a data analysis spec. Use only the tools provided by the principal. Create human- and machine-checkable artifacts. Perform the four design diagnostics in addition to the core analysis. Report any …”
3
Critic evaluates and flags gaps
ai_action
“Critics: Check for blind spots, such as unmentioned confounders, in the principal's plan. Check for alignment between the plan, spec, and executed analysis. Specify a credibility level in the results after seeing the diagnostics. Suggest…”
4
Remediation on diagnostic failure
ai_action
“To address these diagnostic failures, our workflow provides agents with a playbook. For example, to overcome poor overlap, we instruct the agent to use Crump-style trimming”
5
Human reviews published artifacts
human_review
“agents version-control their reports and upload executed notebooks to a file store, where they can be downloaded and re-executed by principals (if they wish)”
Reported outcome

The scaffolded agentic workflow recovered ground truth in nine out of ten ACIC benchmark datasets; the critic agent separated reliable estimates (192 satisfactory, lower RMSE, better-calibrated confidence intervals) from unreliable ones (39 unsatisfactory), and the workflow reduced human toil on iterative causal analyses at Netflix.

Reported metrics
Scaffolded vs unscaffolded baseline estimate25% of the baseline
ACIC ground truth recovery with scaffoldingnine out of ten datasets
satisfactory estimates out of 231 ACIC runs192
unsatisfactory estimates out of 231 ACIC runs39
Show all 6 reported metrics
scaffolded vs unscaffolded baseline estimate25% of the baseline
ACIC ground truth recovery with scaffoldingnine out of ten datasets
satisfactory estimates out of 231 ACIC runs192
unsatisfactory estimates out of 231 ACIC runs39
agent vs one-shot performance on benchmarksystematically beats one-shot iterations
total ACIC evaluation runs231
Reported stack
Claude Sonnet 4.6oci-agentEconML
Source
https://netflixtechblog.com/a-human-augmenting-agentic-workflow-for-causal-inference-4623f0a9c5af
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The scaffolded agentic workflow recovered ground truth in nine out of ten ACIC benchmark datasets; the critic agent separated reliable estimates (192 satisfactory, lower RMSE, better-calibrated confidence intervals) f…

What tools did this team use?

Claude Sonnet 4.6, oci-agent, EconML.

What results were reported?

Scaffolded vs unscaffolded baseline estimate: 25% of the baseline; ACIC ground truth recovery with scaffolding: nine out of ten datasets; satisfactory estimates out of 231 ACIC runs: 192; unsatisfactory estimates out of 231 ACIC runs: 39 (source-reported, not independently verified).

What failed first in this deployment?

One-shot LLM prompting without scaffolding produced consistently wrong answers on benchmark datasets; in the Netflix case study the paved-path agentic workflow produced an updated estimate that was just 25% of the bas…

How is this back office ops AI workflow structured?

Principal submits analysis plan → Actor refines plan and executes analysis → Critic evaluates and flags gaps → Remediation on diagnostic failure → Human reviews published artifacts.

Related back office ops cases