Workflow · Production

Netflix develops Advantage-Weighted Supervised Fine-Tuning (A-SFT) to post-train generative recommender systems on noisy reward signals

The problem

Generative recommenders trained purely by imitating observed user behavior can perpetuate suboptimal recommendations because user interactions are influenced by trends and external suggestions. Standard post-training techniques developed for LLMs (PPO, DPO) cannot be directly applied to recommendation systems due to the absence of counterfactual data, noisy reward models, and unknown logging policies.

First attempt

Reward models trained for recommendation settings do not significantly outperform simple baselines (average user reward or average title reward), because users explore only a small subset of titles and their viewing choices exhibit permutation invariance that makes reward learning difficult.

Workflow diagram · grounded in source
1
User feedback signal collection
trigger
“user feedback, which includes explicit signals such as ratings and reviews, as well as implicit signals like watch time, click-through rates, and overall engagement”
2
Reward model training
ai_action
“This reward model is co-trained as a shallow reward head on top of the generative recommender. It predicts the reward for the most recently selected title based on a user's interaction history.”
3
Advantage-weighted fine-tuning
ai_action
“A-SFT combines supervised fine-tuning with the advantage function to more effectively guide post-training optimization. This approach proves especially effective when the reward model has high variance but still provides valuable directi…”
4
Offline benchmark evaluation
validation
“We sampled a separate test set of O(Millions) users. This test set is collected on a future date after the training.”
5
Recommendation generation
output
“given a history of user interactions as the context, can we recommend a high reward next title recommendation for the user”
Reported outcome

A-SFT achieves better alignment between the pre-trained generative recommendation model and the reward model, outperforming baseline behavior cloning as well as reward-model-dependent algorithms (CQL, PPO, DPO, IPO) on recommendation metrics including NDCG, HR, and MRR.

Reported metrics
A-SFT improvement vs. baseline methodslargest improvement in metrics
Reward model vs. simple baselinesdo not significantly outperform the simple baselines
Reward evaluation standard deviationless than 4%
Reported stack
HSTUPPODPOGRPOCQLIPO
Source
https://netflixtechblog.com/post-training-generative-recommenders-with-advantage-weighted-supervised-finetuning-61a538d717a9
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

A-SFT achieves better alignment between the pre-trained generative recommendation model and the reward model, outperforming baseline behavior cloning as well as reward-model-dependent algorithms (CQL, PPO, DPO, IPO) o…

What tools did this team use?

HSTU, PPO, DPO, GRPO, CQL, IPO.

What results were reported?

A-SFT improvement vs. baseline methods: largest improvement in metrics; Reward model vs. simple baselines: do not significantly outperform the simple baselines; Reward evaluation standard deviation: less than 4% (source-reported, not independently verified).

What failed first in this deployment?

Reward models trained for recommendation settings do not significantly outperform simple baselines (average user reward or average title reward), because users explore only a small subset of titles and their viewing c…

How is this workflow AI workflow structured?

User feedback signal collection → Reward model training → Advantage-weighted fine-tuning → Offline benchmark evaluation → Recommendation generation.