Marketing ops · Production

Spotify uses LLMs to deliver personalized recommendation explanations and AI DJ commentary

The problem

Spotify's recommendation system relied on cover art and artist familiarity, leaving users without context for why a recommendation might resonate with them personally and limiting discovery of new and niche content.

First attempt

Initial zero-shot and few-shot prompting of open-source models revealed challenges with consistent generation style, safety measures, hallucination prevention, and accurate understanding of user preferences.

Workflow diagram · grounded in source

User encounters recommendation

trigger

“Traditionally, Spotify users rely on cover art and familiarity with an artist or genre when deciding whether to engage with music recommendations”

LLM generates recommendation explanation

ai_action

“we explored how LLMs can generate concise explanations that add valuable context to recommendations for music, podcasts, and audiobooks over the past months”

Human editors review and provide feedback

human_review

“Expert editors provided "golden examples" of contextualization. They also provided ongoing feedback to address challenges in LLM output, including artist attribution errors, tone inconsistencies, and factual inaccuracies”

Instruction tuning and adversarial testing

feedback_loop

“we also incorporated targeted prompt engineering, instruction tuning, and scenario-based adversarial testing to generate the recommendation explanations”

AI DJ delivers real-time commentary

ai_action

“DJ is a personalized AI guide that deeply understands listeners' music tastes, providing tailored song selections and insightful commentary on the artists and tracks it recommends”

Personalized narrative delivered to listener

output

“listeners are more willing to listen to a song they may otherwise skip”

Reported outcome

Users were up to four times more likely to click on recommendations with explanations; fine-tuned Llama models achieved up to 14% improvement on Spotify-specific tasks while significantly reducing costs and latency for AI DJ commentary.

Reported metrics

Recommendation click-through rate with explanationsup to four times more likely to click

Spotify-specific task performance improvement vs baseline Llamaup to 14%

AI DJ model costs and latencysignificantly reducing costs and latency

LLM checkpointing timesignificantly reduced checkpointing time

Reported stack

LlamaLlama 3.1 8BvLLMMMLU

Source

https://research.atspotify.com/2024/12/contextualized-recommendations-through-personalized-narratives-using-llms/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Users were up to four times more likely to click on recommendations with explanations; fine-tuned Llama models achieved up to 14% improvement on Spotify-specific tasks while significantly reducing costs and latency fo…

What tools did this team use?

Llama, Llama 3.1 8B, vLLM, MMLU.

What results were reported?

Recommendation click-through rate with explanations: up to four times more likely to click; Spotify-specific task performance improvement vs baseline Llama: up to 14%; AI DJ model costs and latency: significantly reducing costs and latency; LLM checkpointing time: significantly reduced checkpointing time (source-reported, not independently verified).

What failed first in this deployment?

How is this marketing ops AI workflow structured?

User encounters recommendation → LLM generates recommendation explanation → Human editors review and provide feedback → Instruction tuning and adversarial testing → AI DJ delivers real-time commentary → Personalized narrative delivered to listener.