Ecommerce ops · Production

Zalando leverages Multimodal LLMs as judge for large-scale product retrieval evaluation

The problem

Evaluating product search relevance at scale is essential for e-commerce platforms but traditionally relies on human relevance assessments that require substantial time and resources, making large-scale multilingual evaluation impractical.

Workflow diagram · grounded in source
1
Search log query extraction
trigger
“Query extraction: query-product pairs are extracted from search logs for evaluation.”
2
LLM annotation guideline generation
ai_action
“Guideline generation: for each query, an LLM generates custom annotation guidelines, setting detailed criteria for relevance.”
3
Multimodal relevance scoring
ai_action
“Multimodal annotation: MLLMs assign relevancy scores to the search results based on both textual and visual descriptions, classifying each result as "highly relevant", "acceptable substitute", or "irrelevant".”
4
Labeled pair storage
output
“Evaluation and storage: each labeled pair is stored for continuous retrieval system evaluation and comparison across different configurations.”
5
Production monitoring and improvement
feedback_loop
“This framework has been deployed in production at Zalando, enabling regular monitoring of high-frequency search queries and identifying low-performing queries for targeted improvements.”
Reported outcome

The deployed framework achieves relevance assessment quality comparable to human annotations at up to 1,000 times lower cost, evaluating 20,000 query-product pairs in around 20 minutes, and enables continuous production monitoring at Zalando.

Reported metrics
Cost vs human laborup to 1,000 times cheaper than human labor
Evaluation speed for 20,000 query-product pairsaround 20 minutes
Annotation quality vs humancomparable quality to human annotations
Share of hard disagreements caused by human errors50%
Show all 6 reported metrics
cost vs human laborup to 1,000 times cheaper than human labor
evaluation speed for 20,000 query-product pairsaround 20 minutes
annotation quality vs humancomparable quality to human annotations
share of hard disagreements caused by human errors50%
share of hard disagreements caused by LLM errors31%
share of hard disagreements where both made errors19%
Reported stack
GPT-4oGPT-4 TurboGPT-3.5 Turbo
Source
https://engineering.zalando.com/posts/2024/11/llm-as-a-judge-relevance-assessment-paper-announcement.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The deployed framework achieves relevance assessment quality comparable to human annotations at up to 1,000 times lower cost, evaluating 20,000 query-product pairs in around 20 minutes, and enables continuous producti…

What tools did this team use?

GPT-4o, GPT-4 Turbo, GPT-3.5 Turbo.

What results were reported?

Cost vs human labor: up to 1,000 times cheaper than human labor; Evaluation speed for 20,000 query-product pairs: around 20 minutes; Annotation quality vs human: comparable quality to human annotations; Share of hard disagreements caused by human errors: 50% (source-reported, not independently verified).

How is this ecommerce ops AI workflow structured?

Search log query extraction → LLM annotation guideline generation → Multimodal relevance scoring → Labeled pair storage → Production monitoring and improvement.

Related ecommerce ops cases