Back office ops · Production

Netflix optimizes Ranker serendipity scoring CPU by ~7% using JDK Vector API batching

The problem

Netflix's Ranker service had a CPU hotspot in serendipity scoring — the logic that measures how different a candidate title is from a member's viewing history. The original O(M×N) per-pair cosine similarity loop consumed about 7.5% of total CPU per node due to sequential work, repeated embedding lookups, and poor cache locality.

First attempt

An initial batching attempt caused a ~5% performance regression because double[][] matrices created GC pressure and non-contiguous memory hurt cache behavior. A subsequent BLAS integration failed to deliver gains in production due to the F2J fallback, JNI overhead, and a row-major vs. column-major layout mismatch.

Workflow diagram · grounded in source
1
Embedding retrieval
trigger
“A candidate title and each item in a member's viewing history are represented as embeddings in a vector space”
2
Matrix packing and normalization
ai_action
“Pack all candidate embeddings into a matrix A of shape M x D ... Pack all history embeddings into a matrix B of shape N x D ... Normalize all rows to unit length”
3
SIMD matrix multiply
ai_action
“the inner loop computes a dot product by accumulating a * b into a vector accumulator using fma() (fused multiply-add). DoubleVector.SPECIES_PREFERRED lets the runtime pick an appropriate lane width for the machine”
4
Serendipity score derivation
ai_action
“find the maximum similarity, and convert that into a "novelty" score”
5
Feature emission to Ranker
output
“That score becomes an input feature to the downstream recommendation logic”
Reported outcome

With batching, flat buffers, ThreadLocal reuse, and the JDK Vector API in place, Netflix achieved a ~7% drop in CPU utilization, a ~12% drop in average latency, and a ~10% improvement in CPU per request-per-second.
The serendipity encoder's share of CPU fell from 7.5% to ~1%.

Reported metrics
serendipity scoring CPU share (before)7.5%
serendipity scoring CPU share (after)~1%
cluster CPU utilization drop~7%
Average latency drop~12%
Show all 8 reported metrics
serendipity scoring CPU share (before)7.5%
serendipity scoring CPU share (after)~1%
cluster CPU utilization drop~7%
average latency drop~12%
CPU per request-per-second improvementroughly 10%
initial batching performance regression~5%
single-video request share98%
total video volume: single vs batch splitroughly 50:50 between single and batch jobs
Reported stack
JDK Vector APIBLASnetlib-javaTensorFlowLucene
Source
https://netflixtechblog.com/optimizing-recommendation-systems-with-jdks-vector-api-30d2830401ec
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

With batching, flat buffers, ThreadLocal reuse, and the JDK Vector API in place, Netflix achieved a ~7% drop in CPU utilization, a ~12% drop in average latency, and a ~10% improvement in CPU per request-per-second.

What tools did this team use?

JDK Vector API, BLAS, netlib-java, TensorFlow, Lucene.

What results were reported?

serendipity scoring CPU share (before): 7.5%; serendipity scoring CPU share (after): ~1%; cluster CPU utilization drop: ~7%; Average latency drop: ~12% (source-reported, not independently verified).

What failed first in this deployment?

An initial batching attempt caused a ~5% performance regression because double[][] matrices created GC pressure and non-contiguous memory hurt cache behavior.

How is this back office ops AI workflow structured?

Embedding retrieval → Matrix packing and normalization → SIMD matrix multiply → Serendipity score derivation → Feature emission to Ranker.

Related back office ops cases