Back office ops · Production

Netflix builds an internal LLM post-training framework scaling from SFT to on-policy RL

The problem

At Netflix scale, post-training LLMs became an engineering problem as much as a modeling one — researchers had to manage complex data pipelines, distributed GPU clusters, and multi-stage orchestration instead of focusing on model innovation.

First attempt

The original SFT-centric SPMD architecture could not support on-policy RL workflows that emerged with DeepSeek-R1 and GRPO. Separately, binding to low-level tokenization libraries created a silent training-serving token skew that caused inexplicable quality regressions.

Workflow diagram · grounded in source
1
Configuration-driven job submission
trigger
“Users typically express jobs as configuration files that select a recipe and plug in task-specific components”
2
Async on-the-fly sequence packing
ai_action
“we built on-the-fly sequence packing that streams samples from storage and dynamically packs them in memory. Packing runs asynchronously, overlapping CPU work with GPU compute”
3
Distributed training via Ray actors
ai_action
“training runs as a distributed job. We use Ray to orchestrate workflows via actors, decoupling modeling logic from hardware”
4
RL multi-stage rollout and policy update
ai_action
“decompose the system into distinct roles — Policy, Rollout Workers, Reward Model, Reference Model, etc. — and evolve the driver into an active controller that encodes the control plane: when to generate rollouts, how to batch and score t…”
5
AI agents automate architecture conversion
ai_action
“we use AI coding agents to automate much of the conversion work, with a strict logit verifier as the gate: given random inputs, our internal model must match the Hugging Face logits within tolerance. Because the acceptance criterion is m…”
6
Checkpoint for fault resumption
output
“standardized checkpoints to resume cleanly after failures”
Reported outcome

Netflix shipped a managed post-training framework covering SFT, DPO, RL, and Knowledge Distillation, lowering the barrier for teams to iterate on advanced techniques.
On-the-fly sequence packing improved effective token throughput by up to 4.7x for their most skewed dataset.

Reported metrics
Effective token throughput improvementup to 4.7x
Vocabulary head execution time degradation (pre-fix)tripling that layer's execution time
Time-to-support for new architecturesdramatically shortening the time-to-support for new architectures
Reported stack
PyTorchRayvLLMVerlMakoAWSHugging Face AutoTokenizerFSDPLoRAFlexAttention
Source
https://netflixtechblog.com/scaling-llm-post-training-at-netflix-0046f8790194
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Netflix shipped a managed post-training framework covering SFT, DPO, RL, and Knowledge Distillation, lowering the barrier for teams to iterate on advanced techniques.

What tools did this team use?

PyTorch, Ray, vLLM, Verl, Mako, AWS, Hugging Face AutoTokenizer, FSDP, LoRA, FlexAttention.

What results were reported?

Effective token throughput improvement: up to 4.7x; Vocabulary head execution time degradation (pre-fix): tripling that layer's execution time; Time-to-support for new architectures: dramatically shortening the time-to-support for new architectures (source-reported, not independently verified).

What failed first in this deployment?

The original SFT-centric SPMD architecture could not support on-policy RL workflows that emerged with DeepSeek-R1 and GRPO.

How is this back office ops AI workflow structured?

Configuration-driven job submission → Async on-the-fly sequence packing → Distributed training via Ray actors → RL multi-stage rollout and policy update → AI agents automate architecture conversion → Checkpoint for fault resumption.

Related back office ops cases