Data entry ops · Production

How Spotify Generated Millions of ML Content Annotations Using a Scalable Annotation Platform

The problem

Spotify's ML teams needed high-quality annotations at massive scale — covering hundreds of millions of tracks and podcast episodes — but the manual annotation process was inefficient, disconnected, and lacked the right context for engineers and domain experts to operate effectively.

Workflow diagram · grounded in source
1
Annotation case sampling
trigger
“We developed scripts to sample predictions, served data for operator review, and integrated the results with model training and evaluation workflows”
2
LLM parallel annotation
ai_action
“we also built a configurable, LLM-based system that runs in parallel to the human experts. It has allowed us to significantly grow our corpus of high-quality annotation data with low effort and cost”
3
Core annotator first-pass review
human_review
“Core annotator workforces: These workforces are domain experts, who provide first-pass review of all annotation cases”
4
Agreement metric validation
validation
“we started to compute an overall "agreement" metric. Any data points without a clear resolution were automatically escalated to our quality analysts”
5
Quality analyst escalation
human_review
“Quality analysts are top-level domain experts, who act as the escalation point for all ambiguous or complex cases identified by the core annotator workforce”
6
ML workflow integration
integration
“integrated the results with model training and evaluation workflows”
7
High-confidence annotation output
output
“ensures that our models receive the highest confidence annotation for training and evaluation”
Reported outcome

The annotation platform increased the annotation corpus by 10 times and achieved three times the improvement in annotator productivity, while significantly reducing the time it takes to develop new ML models.

Reported metrics
Annotation corpus growth10 times
Annotator productivity improvementthree times
annotation data growth with LLM systemsignificantly grow our corpus of high-quality annotation data with low effort and cost
ML model development timesignificantly reduce the time it takes to develop new ML models
Reported stack
LLM
Source
https://engineering.atspotify.com/2024/10/how-we-generated-millions-of-content-annotations/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The annotation platform increased the annotation corpus by 10 times and achieved three times the improvement in annotator productivity, while significantly reducing the time it takes to develop new ML models.

What tools did this team use?

LLM.

What results were reported?

Annotation corpus growth: 10 times; Annotator productivity improvement: three times; annotation data growth with LLM system: significantly grow our corpus of high-quality annotation data with low effort and cost; ML model development time: significantly reduce the time it takes to develop new ML models (source-reported, not independently verified).

How is this data entry ops AI workflow structured?

Annotation case sampling → LLM parallel annotation → Core annotator first-pass review → Agreement metric validation → Quality analyst escalation → ML workflow integration → High-confidence annotation output.

Related data entry ops cases