Quality assurance · Production

Netflix uses neural embeddings and LSH to compress build logs from millions to thousands of lines

The problem

Netflix engineers faced build logs up to 2.5GB and 3 million lines, making manual bug-finding practically impossible; existing diff tools either produced hundreds of thousands of candidate lines or took an hour to run while still leaving 40,000 lines to review.

First attempt

Standard md5 diff produced hundreds of thousands of candidate lines due to character-level comparison. Fuzzy diffing with k-nearest neighbors took an hour and still yielded 40,000 lines. Neither approach handled semantic similarity between log lines.

Workflow diagram · grounded in source
1
Failed build triggers log diff
trigger
“log entries from a failed software build — 3 million lines — to search for a bug or a regression that happened on line 1M”
2
Lines encoded as embedding vectors
ai_action
“embedding each line into a low dimensional vector and (optionally "fine-tuning" or updating the embedding model at the same time)”
3
LSH cluster assignment
ai_action
“Locality sensitive hashing is a probabilistic algorithm that permits constant time cluster assignment and near-constant time nearest neighbors search”
4
Outlier lines identified
validation
“identifying lines in different clusters as "different"”
5
Compressed outlier report delivered
output
“allows an engineer to look through a small fraction of the log's lines to identify and fix errors in potentially business-critical software”
Reported outcome

Netflix's neural embedding and LSH solution produces 20,000 candidate lines in 20 minutes, enabling engineers to review a small fraction of log output, with examples showing up to 200x log compression.

Reported metrics
Candidate lines in output20,000
Processing time20 min
Log compression ratio200x
Lines reduced (example 1)from 6,892 lines to just 3
Show all 5 reported metrics
candidate lines in output20,000
processing time20 min
log compression ratio200x
lines reduced (example 1)from 6,892 lines to just 3
lines reduced (example 2)6,044 lines, but only 171 were left in the report
Reported stack
Tensorflow 2.2scikit-learn NearestNeighborneural embeddingslocality sensitive hashing
Source
https://netflixtechblog.com/machine-learning-for-a-better-developer-experience-1e600c69f36c
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Netflix's neural embedding and LSH solution produces 20,000 candidate lines in 20 minutes, enabling engineers to review a small fraction of log output, with examples showing up to 200x log compression.

What tools did this team use?

Tensorflow 2.2, scikit-learn NearestNeighbor, neural embeddings, locality sensitive hashing.

What results were reported?

Candidate lines in output: 20,000; Processing time: 20 min; Log compression ratio: 200x; Lines reduced (example 1): from 6,892 lines to just 3 (source-reported, not independently verified).

What failed first in this deployment?

Standard md5 diff produced hundreds of thousands of candidate lines due to character-level comparison.

How is this quality assurance AI workflow structured?

Failed build triggers log diff → Lines encoded as embedding vectors → LSH cluster assignment → Outlier lines identified → Compressed outlier report delivered.