Back office ops · Production

Supporting Diverse ML Systems at Netflix with the Metaflow Platform

The problem

Netflix's diverse ML and AI use cases needed a path from prototype to production without forcing all teams down the same route or maintaining projects as operational outliers with unsustainable overhead.

Workflow diagram · grounded in source
1
Title pairs loaded from warehouse
trigger
“load large amounts of title information — approximately a billion pairs — stored in the Netflix Data Warehouse, so the pairs can be matched in parallel across many Metaflow tasks”
2
Input shards resolved and distributed
integration
“We use metaflow.Table to resolve all input shards which are distributed to Metaflow tasks which are responsible for processing terabytes of data collectively”
3
Parallel entity matching
ai_action
“Each task loads the data using metaflow.MetaflowDataFrame, performs matching using Pandas, and populates a corresponding shard in an output Table”
4
Output table committed
output
“when all matching is done and data is written the new table is committed so it can be read by other jobs”
Reported outcome

Netflix now runs hundreds of Metaflow projects in production, supporting business-critical systems across content, media, and infrastructure, with complex pipelines managed autonomously by relatively small teams.

Reported metrics
Metaflow projects in productionhundreds of Metaflow projects deployed internally
Feature shipping speedship the feature faster with no additional operational burden
GPU cost savings from idle service scale-downsaving cost particularly if your service requires GPUs
Reported stack
MetaflowApache IcebergApache SparkApache ArrowPandasPolarsTitusMaestroStreamlit
Source
https://netflixtechblog.com/supporting-diverse-ml-systems-at-netflix-2d2e6b6d205d
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Netflix now runs hundreds of Metaflow projects in production, supporting business-critical systems across content, media, and infrastructure, with complex pipelines managed autonomously by relatively small teams.

What tools did this team use?

Metaflow, Apache Iceberg, Apache Spark, Apache Arrow, Pandas, Polars, Titus, Maestro, Streamlit.

What results were reported?

Metaflow projects in production: hundreds of Metaflow projects deployed internally; Feature shipping speed: ship the feature faster with no additional operational burden; GPU cost savings from idle service scale-down: saving cost particularly if your service requires GPUs (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Title pairs loaded from warehouse → Input shards resolved and distributed → Parallel entity matching → Output table committed.

Related back office ops cases