Marketing ops · Production

Inside the Archive: How Spotify generated 1.4 billion personalized LLM reports for 2025 Wrapped

The problem

Spotify wanted to identify meaningful listening moments from each user's 2025 history and generate a personalized narrative story for them, at a scale of roughly 350 million eligible users each receiving up to five reports, totaling approximately 1.4 billion LLM-generated reports — a volume that made high-performance models economically infeasible.

First attempt

A timezone bug in the upstream data pipeline caused some Biggest Discovery Day reports to celebrate the wrong number of artists discovered; the LLM faithfully generated a compelling but factually incorrect story from the flawed data.

Workflow diagram · grounded in source
1
Heuristic remarkable day identification
ai_action
“To identify the "remarkable days" per user from an entire year of listening, we designed a priority-ordered set of heuristics. By ranking these candidates in order of narrative potential and statistical strength, we narrowed hundreds of …”
2
Distributed pipeline and queue publish
integration
“We used a distributed data pipeline to compute and aggregate candidate days at the user level. For each user, we stored their remarkable days and relevant listening history data to object storage. When it was time to pre-generate reports…”
3
Fine-tuned LLM report generation
ai_action
“each remarkable day was processed, generating one report at a time per user, so earlier reports could inform later ones to avoid repetition”
4
LLM-as-judge automated evaluation
validation
“Evaluation was performed by larger models acting as judges (LLM as a judge). Each report was graded across four dimensions: accuracy, safety, tone, and formatting.”
5
Human review and brand feedback
human_review
“We layered in human review. Creative, technical, and safety feedback all fed into the next iteration.”
6
Evaluation-driven remediation loop
feedback_loop
“Evaluation fed directly into a structured remediation loop. We identified problematic reports through model-based evaluators and targeted human review, then used SQL queries and regex-based pattern matching to surface structurally simila…”
Reported outcome

Wrapped Archive reached hundreds of millions of users globally; the generation engine ran for four days straight producing roughly 1.4 billion reports, and the structured evaluation and remediation loop caught the timezone bug, fixed the pipeline, and allowed bulk replay of affected reports.

Reported metrics
Total reports pre-generatedroughly 1.4 billion reports
Eligible usersAround 350 million eligible users
Reports sampled for evaluation~165,000 reports
Generation engine runtimefour days
Show all 6 reported metrics
total reports pre-generatedroughly 1.4 billion reports
eligible usersAround 350 million eligible users
reports sampled for evaluation~165,000 reports
generation engine runtimefour days
generation throughputthousands of requests per second
global user reachhundreds of millions of users
Reported stack
fine-tuned modelfrontier modelLLMDirect Preference Optimization (DPO)pubsub message queueevaluation data warehousedistributed, column-oriented key-value databaseAI coding assistants
Source
https://engineering.atspotify.com/2026/3/inside-the-archive-2025-wrapped
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Wrapped Archive reached hundreds of millions of users globally; the generation engine ran for four days straight producing roughly 1.4 billion reports, and the structured evaluation and remediation loop caught the tim…

What tools did this team use?

fine-tuned model, frontier model, LLM, Direct Preference Optimization (DPO), pubsub message queue, evaluation data warehouse, distributed, column-oriented key-value database, AI coding assistants.

What results were reported?

Total reports pre-generated: roughly 1.4 billion reports; Eligible users: Around 350 million eligible users; Reports sampled for evaluation: ~165,000 reports; Generation engine runtime: four days (source-reported, not independently verified).

What failed first in this deployment?

A timezone bug in the upstream data pipeline caused some Biggest Discovery Day reports to celebrate the wrong number of artists discovered; the LLM faithfully generated a compelling but factually incorrect story from…

How is this marketing ops AI workflow structured?

Heuristic remarkable day identification → Distributed pipeline and queue publish → Fine-tuned LLM report generation → LLM-as-judge automated evaluation → Human review and brand feedback → Evaluation-driven remediation loop.

Related marketing ops cases