Quality assurance · Production

Pinterest Search scales relevance evaluation with fine-tuned LLMs, reducing minimum detectable effects by an order of magnitude

The problem

Pinterest Search's relevance measurement relied on costly, slow human annotations that constrained sample sizes, making it impossible to detect heterogeneous treatment effects or small topline metric changes in A/B experiments.

Workflow diagram · grounded in source

Fine-tune LLM on human labels

ai_action

“We fine-tune open-source LLMs on relevance prediction tasks using human-annotated labels”

Stratified query sampling

trigger

“we take a stratified sample of paired search queries from control and treatment experiment groups, ensuring that the sample is representative of overall user usage”

LLM relevance labeling

ai_action

“we retain the top K search results and generate LLM-based relevance labels”

Compute sDCG@K metrics

output

“We then compute sDCG@K for each query and aggregate query-level metrics to derive topline experiment metrics”

Heterogeneous effects analysis

output

“we calculate heterogeneous effects by query popularity and query interest (e.g. beauty, women's fashion, art, etc), utilizing a Benjamini-Hochberg procedure to control the false discovery rate”

Reported outcome

Fine-tuned LLMs replaced costly human labeling at scale, reducing MDEs to ≤ 0.25% (an order of magnitude reduction) and enabling 150,000 rows to be labeled within 30 minutes on a single GPU, while significantly cutting annotation costs and turnaround time.

Reported metrics

MDE before LLM labeling1.3%-1.5%

MDE after LLM labeling≤ 0.25%

Labeling throughput150,000 rows within 30 minutes

LLM-human exact match rate73.7%

Show all 8 reported metrics

MDE before LLM labeling1.3%-1.5%

MDE after LLM labeling≤ 0.25%

Labeling throughput150,000 rows within 30 minutes

LLM-human exact match rate73.7%

LLM-human ratings within 1 point91.7%

MDE reduction magnitudeorder of magnitude reduction in MDEs

Labeling cost reductionsignificantly reduces labeling costs

Llama-3-8B inference cost vs XLM-RoBERTa-large6 times

Reported stack

XLM-RoBERTa-largeBLIPDistilBERTLlama-3–8Bmultilingual BERT-baseT5-basemDeBERTa-V3-base

Source

https://medium.com/pinterest-engineering/llm-powered-relevance-assessment-for-pinterest-search-b846489e358d

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Fine-tuned LLMs replaced costly human labeling at scale, reducing MDEs to ≤ 0.25% (an order of magnitude reduction) and enabling 150,000 rows to be labeled within 30 minutes on a single GPU, while significantly cuttin…

What tools did this team use?

XLM-RoBERTa-large, BLIP, DistilBERT, Llama-3–8B, multilingual BERT-base, T5-base, mDeBERTa-V3-base.

What results were reported?

MDE before LLM labeling: 1.3%-1.5%; MDE after LLM labeling: ≤ 0.25%; Labeling throughput: 150,000 rows within 30 minutes; LLM-human exact match rate: 73.7% (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Fine-tune LLM on human labels → Stratified query sampling → LLM relevance labeling → Compute sDCG@K metrics → Heterogeneous effects analysis.