Quality assurance · Production

DoorDash builds AutoEval: LLM-powered automated search relevance evaluation at scale

The problem

DoorDash's search quality evaluation relied on human annotation that could not scale: annotation cycles took days or weeks, individual raters interpreted guidelines differently causing label noise, and datasets overrepresented high-frequency queries while underrepresenting tail queries where relevance problems hide.

Workflow diagram · grounded in source
1
Query sampling from live traffic
trigger
“We sample real user queries from live traffic across intent, frequency, geographic, and daypart dimensions.”
2
Structured prompt construction
integration
“Each query-result pair is converted into a structured prompt tailored to the evaluation task such as dish-to-store or cuisine-to-store.”
3
LLM relevance judgment
ai_action
“The prompt is passed to an LLM, base or fine-tuned, which returns a structured relevance judgment.”
4
WPR score aggregation
output
“Judgments are aggregated to generate a page-level WPR score.”
5
Human audit of LLM outputs
human_review
“external raters review a sample of LLM-generated judgments, flagging low-quality outputs which internal experts then investigate”
6
Expert prompt and label refinement
feedback_loop
“Experts analyze flagged outputs and refine prompts or labels.”
Reported outcome

AutoEval reduced relevance judgment turnaround time by 98% compared to human evaluation and unlocked a nine-fold increase in capacity, while fine-tuned LLMs consistently match or outperform external raters in key relevance tasks.
Expert raters were freed from repetitive labeling to focus on guideline development and edge case resolution.

Reported metrics
Relevance judgment turnaround time98% reduction
Evaluation capacitynine-fold increase in capacity
LLM vs external rater accuracyconsistently match or outperform external raters
LLM vs crowd annotator performanceoutperforming crowd annotators in key categories
Show all 6 reported metrics
relevance judgment turnaround time98% reduction
evaluation capacitynine-fold increase in capacity
LLM vs external rater accuracyconsistently match or outperform external raters
LLM vs crowd annotator performanceoutperforming crowd annotators in key categories
automated relevance judgments throughputmillions of relevance judgments per day
offline experimentation pipeline bottleneckresolving a major bottleneck
Reported stack
LLMsAutoEvalOpenAI
Source
https://careersatdoordash.com/blog/doordash-llms-to-evaluate-search-result-pages/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

AutoEval reduced relevance judgment turnaround time by 98% compared to human evaluation and unlocked a nine-fold increase in capacity, while fine-tuned LLMs consistently match or outperform external raters in key rele…

What tools did this team use?

LLMs, AutoEval, OpenAI.

What results were reported?

Relevance judgment turnaround time: 98% reduction; Evaluation capacity: nine-fold increase in capacity; LLM vs external rater accuracy: consistently match or outperform external raters; LLM vs crowd annotator performance: outperforming crowd annotators in key categories (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Query sampling from live traffic → Structured prompt construction → LLM relevance judgment → WPR score aggregation → Human audit of LLM outputs → Expert prompt and label refinement.

Related quality assurance cases