Ecommerce ops · Production

Pinterest improves search relevance using LLM-based teacher-student distillation pipeline

The problem

Pinterest Search relied on engagement signals for ranking, but needed a genuine relevance model to ensure displayed content was pertinent to user queries rather than driven by past behaviour. The system also lacked coverage for multilingual queries and seasonal new concepts not found in limited human-annotated data.

First attempt

The LLM-based cross-encoder teacher model was effective at predicting relevance but could not be deployed directly for real-time serving due to latency and cost constraints.

Workflow diagram · grounded in source
1
User search query received
trigger
“Pinterest Search is one of the key surfaces on Pinterest where users can discover inspiring content that aligns with their information needs”
2
Enrich Pin text via BLIP captions
ai_action
“synthetic image descriptions generated by Bootstrapping Language-Image Pre-training (BLIP), an off-the-shelf image captioning model”
3
LLM teacher classifies query-Pin relevance
ai_action
“We use a cross-encoder language model to predict a Pin's relevance to a query, along with Pin text, as shown in Figure 1. The task is formulated as a multiclass classification problem.”
4
Distill teacher labels to student model
ai_action
“we employ the LLM-based teacher model to generate 5-scale relevance labels on a daily logged large search engagement and impression dataset with billions of rows. This labeled dataset is subsequently used to train the much smaller studen…”
5
Student model determines search ranking
output
“The relevance scores generated by the student model are then utilized alongside engagement predictions to determine the final ranking of search results.”
Reported outcome

The LLM-based relevance pipeline achieved a +2.18% improvement in search feed relevance measured by nDCG@20, and online A/B experiments showed improvements of more than 1% in search feed relevance and more than 1.5% in search fulfillment rates.
The multilingual teacher also generalised to languages not seen during training.

Reported metrics
search feed relevance improvement (nDCG@20, human eval)+2.18%
search feed relevance (online A/B experiment)>1%
search fulfillment rates (online A/B experiment)>1.5%
Llama-3-8B vs BERT-base 5-scale accuracy gain12.5%
Show all 5 reported metrics
search feed relevance improvement (nDCG@20, human eval)+2.18%
search feed relevance (online A/B experiment)>1%
search fulfillment rates (online A/B experiment)>1.5%
Llama-3-8B vs BERT-base 5-scale accuracy gain12.5%
Llama-3-8B vs baseline 5-scale accuracy gain19.7%
Reported stack
Llama-3-8BBLIPqLoRABM25mDeBERTa-V3-baseXLM-RoBERTa-largemultilingual BERT-baseT5-baseHugging Face
Source
https://medium.com/pinterest-engineering/improving-pinterest-search-relevance-using-large-language-models-4cd938d4e892
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM-based relevance pipeline achieved a +2.18% improvement in search feed relevance measured by nDCG@20, and online A/B experiments showed improvements of more than 1% in search feed relevance and more than 1.5% i…

What tools did this team use?

Llama-3-8B, BLIP, qLoRA, BM25, mDeBERTa-V3-base, XLM-RoBERTa-large, multilingual BERT-base, T5-base, Hugging Face.

What results were reported?

search feed relevance improvement (nDCG@20, human eval): +2.18%; search feed relevance (online A/B experiment): >1%; search fulfillment rates (online A/B experiment): >1.5%; Llama-3-8B vs BERT-base 5-scale accuracy gain: 12.5% (source-reported, not independently verified).

What failed first in this deployment?

The LLM-based cross-encoder teacher model was effective at predicting relevance but could not be deployed directly for real-time serving due to latency and cost constraints.

How is this ecommerce ops AI workflow structured?

User search query received → Enrich Pin text via BLIP captions → LLM teacher classifies query-Pin relevance → Distill teacher labels to student model → Student model determines search ranking.

Related ecommerce ops cases