Ecommerce ops · Production

DoorDash builds calibrated LLM-as-a-judge to evaluate natural language search quality at scale

The problem

DoorDash's natural language search pipeline accepted vague, intent-based queries with no historical click-through ground truth, making evaluation impossible with traditional search metrics. Human annotation cycles took 2–5 days, produced inconsistent labels for compositional queries due to rubric ambiguity, and a supervised relevance model trained on those labels performed barely above chance.

First attempt

A Qwen3-based reranker trained on human-annotated labels achieved only AUC 0.56 on a held-out set. A systematic audit found that 19 of 35 manually reviewed cases had incorrect human ratings after cross-functional adjudication, with disagreement exceeding 30% on the boundary cases most critical for ranking model training.

Workflow diagram · grounded in source
1
Natural language query submitted
trigger
“These queries describe a situation or a product with multiple requirements. The user isn't naming a cuisine or a store; they're expressing an intent that requires the search engine to understand meaning, not match tokens.”
2
LLM rewrites intent to structured query
ai_action
“It's an LLM-powered pipeline that rewrites vague intent into structured retrieval queries, retrieves candidates via semantic embeddings, and ranks results by relevance.”
3
Binary facet rubric defined, golden set labeled
human_review
“Product managers, the strategy and operations team, and engineers collaboratively redesigned the rubric by decomposing single vague questions into independent binary checks, each targeting one constraint, answerable from observable menu …”
4
LLM judge calibrated against golden set
validation
“The LLM judge grades the same golden set. We measure alignment against adjudicated human consensus, not individual rater labels, which we've established are noisy. When the judge diverges, we iterate by adjusting the prompt, enriching th…”
5
Production monitoring dashboard
output
“Live search traces flow through daily LLM judge grading. Results populate a quality dashboard with standardized metrics such as NDCG@5 or per-facet precision to give the team a daily read on search quality without waiting for human evalu…”
6
PR-level regression guardrail
feedback_loop
“When an engineer opens a pull request that touches the search pipeline, the system deploys the change to a sandbox, replays a standard query set, and runs side-by-side comparison between control and treatment. The LLM judge provides comp…”
Reported outcome

DoorDash replaced periodic human annotation with a calibrated LLM judge running continuously in production monitoring and as a PR-level quality gate, enabling per-facet evaluation that surfaced real performance gaps previously invisible in aggregate NDCG scores.

Reported metrics
Qwen3-based reranker AUC on human-labeled held-out set0.56
Human annotation cases with incorrect ratings (out of 35 reviewed)19 of 35
Inter-annotator disagreement on boundary relevance casesexceeds 30%
Human annotation cycle turnaround timetwo to five days
Show all 7 reported metrics
Qwen3-based reranker AUC on human-labeled held-out set0.56
Human annotation cases with incorrect ratings (out of 35 reviewed)19 of 35
Inter-annotator disagreement on boundary relevance casesexceeds 30%
Human annotation cycle turnaround timetwo to five days
Duplicate store results across queries for same consumer40%
NDCG@5 for discovery/indecision queries0.743
NDCG@5 for vibe/mood queries0.844
Reported stack
o3-miniQwen3semantic embeddings
Source
https://careersatdoordash.com/blog/doordash-llm-as-a-judge-evaluating-natural-language-search/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

DoorDash replaced periodic human annotation with a calibrated LLM judge running continuously in production monitoring and as a PR-level quality gate, enabling per-facet evaluation that surfaced real performance gaps p…

What tools did this team use?

o3-mini, Qwen3, semantic embeddings.

What results were reported?

Qwen3-based reranker AUC on human-labeled held-out set: 0.56; Human annotation cases with incorrect ratings (out of 35 reviewed): 19 of 35; Inter-annotator disagreement on boundary relevance cases: exceeds 30%; Human annotation cycle turnaround time: two to five days (source-reported, not independently verified).

What failed first in this deployment?

A Qwen3-based reranker trained on human-annotated labels achieved only AUC 0.56 on a held-out set.

How is this ecommerce ops AI workflow structured?

Natural language query submitted → LLM rewrites intent to structured query → Binary facet rubric defined, golden set labeled → LLM judge calibrated against golden set → Production monitoring dashboard → PR-level regression guardrail.

Related ecommerce ops cases