Marketing ops · Production

MediaFM: Netflix's Tri-Modal AI Foundation Model for Media Understanding

The problem

Netflix needed scalable machine-level understanding of its entire content catalog — including new formats like live events and podcasts — to power recommendations, ad relevancy, and promotional asset optimization, all of which require sophisticated long-form video understanding of narrative dependencies and emotional arcs spanning entire episodes or films.

First attempt

Prior models not leveraging the full multimodal signal failed to grasp content essence, and the page's ablations show that using multiple modalities without contextualization can actually hurt performance on tasks like clip popularity ranking.

Workflow diagram · grounded in source
1
Shot boundary segmentation
trigger
“The model's fundamental unit of input is a shot, derived by segmenting a movie or episode (collectively referred to as "title") using a shot boundary detection algorithm”
2
Tri-modal embedding extraction
ai_action
“For each shot, we generate three distinct embeddings from its core modalities”
3
Fused vector construction
ai_action
“the three embeddings² are concatenated and unit-normed to form a single 2304-dimensional fused embedding vector”
4
Transformer shot contextualization
ai_action
“The core of our model is a transformer encoder, architecturally similar to BERT. A sequence of preprocessed shot embeddings is passed through the following stages”
5
Downstream task inference
ai_action
“To evaluate the learned embeddings, we learn task-specific linear layers on top of frozen representations (i.e., linear probes)”
6
Ad serving and recommendations integration
integration
“these representations operate at the retrieval stage, where they help in identifying the candidate set and in turn are fed into the ad serving system for relevance optimization”
Reported outcome

MediaFM outperforms all baselines on all evaluated tasks, with clip retrieval improving by around 15% at each model enhancement step, and with larger gains on tasks requiring detailed narrative understanding such as ad relevancy.

Reported metrics
Clip retrieval improvement per model enhancement steparound 15%
MediaFM vs baselines across all tasksbetter than the baselines on all tasks
optimizer switch to Muonnoticeable improvements
Reported stack
SeqCLIPwav2vec2text-embedding-3-largeMuonAdamW
Source
https://netflixtechblog.com/mediafm-the-multimodal-ai-foundation-for-media-understanding-at-netflix-e8c28df82e2d
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

MediaFM outperforms all baselines on all evaluated tasks, with clip retrieval improving by around 15% at each model enhancement step, and with larger gains on tasks requiring detailed narrative understanding such as a…

What tools did this team use?

SeqCLIP, wav2vec2, text-embedding-3-large, Muon, AdamW.

What results were reported?

Clip retrieval improvement per model enhancement step: around 15%; MediaFM vs baselines across all tasks: better than the baselines on all tasks; optimizer switch to Muon: noticeable improvements (source-reported, not independently verified).

What failed first in this deployment?

Prior models not leveraging the full multimodal signal failed to grasp content essence, and the page's ablations show that using multiple modalities without contextualization can actually hurt performance on tasks lik…

How is this marketing ops AI workflow structured?

Shot boundary segmentation → Tri-modal embedding extraction → Fused vector construction → Transformer shot contextualization → Downstream task inference → Ad serving and recommendations integration.

Related marketing ops cases