Zalando builds LLM-as-a-judge search quality assurance framework for multi-market launches
Zalando's pre-launch search quality assurance relied entirely on human experts manually sampling and translating queries, annotating errors, and diagnosing root causes. The process was not scalable and was reactive by nature — issues were only caught after launch when real-user signals such as CTR existed. For entirely new markets, those signals did not exist at all.
The LLM-as-a-judge evaluation framework identified multiple NER and search quality issues in Portuguese and Greek markets before go-live, enabling engineers to fix them pre-launch.
A full run covers 1,500 search segments with 25 results each, completes in 3-5 hours, and costs around 250 USD — compared to days of human evaluation.
Show all 6 reported metrics
Frequently asked questions
What did this team achieve with this AI workflow?
The LLM-as-a-judge evaluation framework identified multiple NER and search quality issues in Portuguese and Greek markets before go-live, enabling engineers to fix them pre-launch.
What tools did this team use?
GPT-4o, Apache Airflow, Kubernetes, Nakadi, Elasticache, NER.
What results were reported?
Cost per evaluation run: around 250 USD; Search segments evaluated per run: 1,500; Search results evaluated per query: 25; Evaluation run duration: 3-5 hours (source-reported, not independently verified).
How is this quality assurance AI workflow structured?
Historical query clustering by NER → LLM query translation → Search result retrieval → LLM-as-a-judge relevance scoring → NER tag consistency validation → Issue flagging and report output → Engineer investigation and fix.