Back office ops · Production

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

The problem

As Netflix's ML investments scaled across business domains, models became black boxes with no discovery infrastructure — practitioners had to traverse fragmented, siloed tools to answer basic questions about lineage, ownership, and impact, and cross-domain reuse of ML assets was extraordinarily difficult.

First attempt

The siloed tooling left each system unaware of the others — the model registry did not know which A/B tests used its models, the pipeline orchestrator was unaware of downstream model dependencies, and practitioners had no way to answer cross-domain impact or lineage questions.

Workflow diagram · grounded in source
1
Source system event ingestion
trigger
“MDS integrates with various source systems via Kafka and AWS SNS/SQS, consuming events in real-time. Source systems emit thin events that include an identifier and an event type”
2
Entity hydration from source APIs
integration
“When an event arrives, MDS: Validates the event schema; Calls the source system's API to fetch the complete, current state; Transforms the response into a normalized entity. This design has a crucial property: the order of events doesn't…”
3
Normalization to unified entity model
integration
“Raw events are heterogeneous and each source system has its own schema and semantics. MDS workers transform these events into a unified entity model with standardized fields”
4
Persist to Datomic and index in Elasticsearch
output
“Once normalized, entities are persisted to Datomic and immediately indexed in Elasticsearch. This happens synchronously within the event processing flow”
5
Background relationship enrichment
integration
“scheduled background processes take over to discover and materialize relationships. These enrichment jobs run periodically, scanning for uncached or partially resolved entities”
6
AIP Portal graph exploration
output
“The Model Lifecycle Graph is surfaced to practitioners through the AIP Portal, a unified interface that provides full-text search across all entity types, detailed entity pages with navigable relationships, and personalized views for tea…”
Reported outcome

Netflix built MDS (Metadata Service) with the Model Lifecycle Graph, enabling every ML practitioner to discover, understand, and reuse ML assets across all domains through the AIP Portal, replacing multi-system manual investigation with single graph queries.

Reported metrics
Relationship materialization delaytypically minutes rather than seconds
Cross-domain query simplificationsingle query
Reported stack
KafkaAWS SNS/SQSDatomicElasticsearch
Source
https://netflixtechblog.com/democratizing-machine-learning-at-netflix-building-the-model-lifecycle-graph-5cc6d5828bb1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Netflix built MDS (Metadata Service) with the Model Lifecycle Graph, enabling every ML practitioner to discover, understand, and reuse ML assets across all domains through the AIP Portal, replacing multi-system manual…

What tools did this team use?

Kafka, AWS SNS/SQS, Datomic, Elasticsearch.

What results were reported?

Relationship materialization delay: typically minutes rather than seconds; Cross-domain query simplification: single query (source-reported, not independently verified).

What failed first in this deployment?

The siloed tooling left each system unaware of the others — the model registry did not know which A/B tests used its models, the pipeline orchestrator was unaware of downstream model dependencies, and practitioners ha…

How is this back office ops AI workflow structured?

Source system event ingestion → Entity hydration from source APIs → Normalization to unified entity model → Persist to Datomic and index in Elasticsearch → Background relationship enrichment → AIP Portal graph exploration.