Back office ops · Production

Netflix builds a Media Understanding Platform for ML-powered dialogue, visual, and shot search across the content catalog

The problem

Netflix artists and video editors spent excessive hours on manual pre-work — watching titles start-to-finish to transcribe dialogue with timecodes (watchdowns) and scrubbing footage to find visual elements. Early ML integration systems were bespoke, tightly coupled, and could not scale across algorithms or teams.

First attempt

Two prior approaches — on-demand batch processing and an online pre-computation system — exposed fundamental scaling problems: disparate systems built by separate teams on different stacks, expensive maintenance, and a tightly coupled architecture that mixed ML algorithms with backend and UI code.

Workflow diagram · grounded in source
1
Editor submits search query
trigger
“Users have flexibility to perform multimodal search with input being a simple text term, image or short video”
2
Query embedding transformation
ai_action
“For queries against embedding based data sources it transforms the input such as text or image to corresponding vector representation. Each data source or algorithm could use a different encoding technique so, this stage ensures that the…”
3
Non-English query translation
ai_action
“All of the text-based models in the platform are trained using English language so we translate non-English text to English”
4
Query routing to searcher
routing
“First we need to federate which query should be routed to which system. This is handled by the Query router and Searcher-proxy module”
5
ML search execution via Marken
ai_action
“The primary searcher used in the current implementation is called Marken — scalable annotation service built at Netflix. It supports different categories of searches including full text and embedding vector based similarity searches”
6
Results ranking and post-processing
ai_action
“It can rank results by applying custom scoring, populate search recommendations based on other similar searches”
7
Paginated results returned
output
“we allow users to paginate the results (with configurable page size) instead of fetching all or a fixed number of results”
Reported outcome

The Media Search Platform (MSP) enables studio creators to find dialogue, visual elements, and similar shots across the Netflix catalog in seconds, while engineers can onboard new ML algorithms independently through a modular, pluggable abstraction layer.

Reported metrics
Time to locate shots (before vs after)what normally would have taken 1–2 people hours/a full day to do, done in seconds
Watchdown creative time wastedwaste countless hours of creative time
Reported stack
MarkenCassandraElasticsearchgRPCGraphQLDomain Graph Service Framework
Source
https://netflixtechblog.com/building-a-media-understanding-platform-for-ml-innovations-9bef9962dcb7
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The Media Search Platform (MSP) enables studio creators to find dialogue, visual elements, and similar shots across the Netflix catalog in seconds, while engineers can onboard new ML algorithms independently through a…

What tools did this team use?

Marken, Cassandra, Elasticsearch, gRPC, GraphQL, Domain Graph Service Framework.

What results were reported?

Time to locate shots (before vs after): what normally would have taken 1–2 people hours/a full day to do, done in seconds; Watchdown creative time wasted: waste countless hours of creative time (source-reported, not independently verified).

What failed first in this deployment?

Two prior approaches — on-demand batch processing and an online pre-computation system — exposed fundamental scaling problems: disparate systems built by separate teams on different stacks, expensive maintenance, and…

How is this back office ops AI workflow structured?

Editor submits search query → Query embedding transformation → Non-English query translation → Query routing to searcher → ML search execution via Marken → Results ranking and post-processing → Paginated results returned.

Related back office ops cases