Netflix's Axion ML Fact Store eliminates training-serving skew and reduces offline feature regeneration from weeks to hours
Netflix's ML models train on weeks of historical data, so testing updated feature encoders required waiting weeks for feature logging to accumulate sufficient data — making experimentation slow and creating training-serving skew risk.
Feature logging required weeks of waiting for data. ETL with normalized multi-table storage caused Spark shuffle issues at scale. Even a single denormalized Iceberg table was too slow for queries filtering hundreds of millions of rows to under a million, and bloom filters plus predicate pushdown were insufficient.
Axion reduces offline feature regeneration from weeks to hours, EVCache queries run 3x–50x faster than Iceberg, and data quality monitoring detects more than 95% of data issues early — making Axion the de facto fact store for Netflix's Personalization ML models.
Frequently asked questions
What did this team achieve with this AI workflow?
Axion reduces offline feature regeneration from weeks to hours, EVCache queries run 3x–50x faster than Iceberg, and data quality monitoring detects more than 95% of data issues early — making Axion the de facto fact s…
What tools did this team use?
Axion, Keystone, Iceberg, EVCache, Spark, Parquet, protobuf, grpc.
What results were reported?
Offline feature regeneration time: hours compared to weeks; EVCache query speed vs Iceberg: 3x-50x faster; Data issues identified early: more than 95% (source-reported, not independently verified).
What failed first in this deployment?
Feature logging required weeks of waiting for data.
How is this back office ops AI workflow structured?
Production inference runs → Facts logged to Keystone → ETL into Iceberg table → EVCache low-latency query tier → Offline feature regeneration → Data quality monitoring.