Quality assurance · Production

Netflix evaluates show synopses at scale using LLM-as-a-Judge, achieving 85%+ agreement with creative writers

The problem

Netflix hosts hundreds of thousands of synopses—often with multiple variants per show—making manual quality validation impossible at scale, while poor synopses directly drive member abandonment.

First attempt

An initial approach of using a single prompt to evaluate all quality criteria overloaded the LLM and yielded poor performance; early human calibration also showed low instance-level agreement due to the subjectivity of the task.

Workflow diagram · grounded in source

Synopsis quality evaluation triggered

trigger

“We host hundreds of thousands of synopses, usually with multiple variants per show. We need to ensure quality at scale”

Per-criteria LLM judging

ai_action

“our automatic evaluation system uses a combination of standard LLM-as-a-Judge, tiered rationales, consensus scoring, and Agents-as-a-Judge to maximize binary scoring accuracy for each criteria”

Factuality agent evaluation

ai_action

“each agent evaluates one narrow aspect of factuality”

Score and rationale aggregation

output

“The final score of the Agents-as-a-Judge system is the minimum factuality score across agents — any failed aspect yields an overall fail. All rationales are fed to an LLM aggregator to produce a combined rationale to accompany the final …”

Streaming metric correlation

feedback_loop

“higher LLM judge quality is correlated with key streaming metrics, allowing us to proactively identify and fix impactful issues weeks or months before a show debuts on Netflix”

Reported outcome

The final LLM-as-a-Judge system achieves 85%+ agreement with creative writers, and its scores correlate with key streaming metrics, enabling proactive identification and fixing of quality issues weeks or months before a show debuts, with widespread adoption in the Netflix synopsis authoring workflow.

Reported metrics

Agreement with creative writers85%+

Human writer calibration agreement~80%

Tone evaluator binary accuracy with tiered rationales87.85%

Tone evaluator binary accuracy baseline86.55%

Reported stack

LLMAgents-as-a-Judge

Source

https://netflixtechblog.com/evaluating-netflix-show-synopses-with-llm-as-a-judge-6269251e6f28

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The final LLM-as-a-Judge system achieves 85%+ agreement with creative writers, and its scores correlate with key streaming metrics, enabling proactive identification and fixing of quality issues weeks or months before…

What tools did this team use?

LLM, Agents-as-a-Judge.

What results were reported?

Agreement with creative writers: 85%+; Human writer calibration agreement: ~80%; Tone evaluator binary accuracy with tiered rationales: 87.85%; Tone evaluator binary accuracy baseline: 86.55% (source-reported, not independently verified).

What failed first in this deployment?

How is this quality assurance AI workflow structured?

Synopsis quality evaluation triggered → Per-criteria LLM judging → Factuality agent evaluation → Score and rationale aggregation → Streaming metric correlation.