Back office ops · Production

Dropbox Dash scales search relevance labeling with LLM-human hybrid pipeline

The problem

Training Dash's search ranking model required high-quality relevance labels at scale, but human labeling was expensive, inconsistent, unable to access sensitive customer data, and impractical at the volumes needed.

First attempt

Neither approach was sufficient alone: human labeling could not scale to the volumes required, and LLMs required careful human calibration before generating reliable relevance judgments. Using LLMs at query time was also infeasible due to latency and context window constraints.

Workflow diagram · grounded in source
1
User submits search query
trigger
“When a user submits a query, Dash first interprets the underlying information need and determines how to retrieve relevant content.”
2
Candidate retrieval and ranking
ai_action
“Search results in Dash are ordered by a relevance model that assigns a score to each document based on how well it matches the query. Like most modern ranking systems, this model is trained rather than hand-tuned.”
3
LLM generates grounded answer
ai_action
“a large language model (LLM) analyzes the most relevant results to generate an answer”
4
Human evaluators label reference set
human_review
“A small group of human evaluators labels a dataset that is orders of magnitude smaller than what would be required for full training. These labels are used to tune the LLM prompt and model parameters. (Human review is conducted by Dropbo…”
5
LLM calibrated against human judgments
validation
“Dash compares LLM-generated relevance ratings with human judgments, rewarding exact matches on a 1–5 relevance scale and applying penalties for disagreement”
6
Error-biased document sampling
ai_action
“Training samples are biased toward situations where mistakes are more likely, since these offer the greatest opportunity for learning. Dash identifies such cases by analyzing discrepancies between user behavior and LLM-predicted relevance.”
7
LLM generates relevance labels at scale
ai_action
“the LLM is deployed to generate hundreds of thousands—or even millions—of relevance labels used to train Dash's relevance model”
8
Prompt optimization with DSPy
feedback_loop
“DSPy can automatically refine prompts to better match human judgments. This makes it possible to reuse the same optimization approach across different evaluation tasks and model configurations”
Reported outcome

By combining a small human-labeled reference set with LLM-based evaluation and iterative prompt optimization via DSPy, Dropbox now generates hundreds of thousands to millions of relevance labels to train Dash's ranking model, with measurable MSE improvement over time.

Reported metrics
Relevance labels generatedhundreds of thousands—or even millions—of relevance labels
LLM relevance evaluator MSEimproved over time
Human labeling dataset size vs full training requirementorders of magnitude smaller
Reported stack
Dropbox DashLLMXGBoostDSPy
Source
https://dropbox.tech/machine-learning/llm-human-labeling-improving-search-relevance-dropbox-dash
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By combining a small human-labeled reference set with LLM-based evaluation and iterative prompt optimization via DSPy, Dropbox now generates hundreds of thousands to millions of relevance labels to train Dash's rankin…

What tools did this team use?

Dropbox Dash, LLM, XGBoost, DSPy.

What results were reported?

Relevance labels generated: hundreds of thousands—or even millions—of relevance labels; LLM relevance evaluator MSE: improved over time; Human labeling dataset size vs full training requirement: orders of magnitude smaller (source-reported, not independently verified).

What failed first in this deployment?

Neither approach was sufficient alone: human labeling could not scale to the volumes required, and LLMs required careful human calibration before generating reliable relevance judgments.

How is this back office ops AI workflow structured?

User submits search query → Candidate retrieval and ranking → LLM generates grounded answer → Human evaluators label reference set → LLM calibrated against human judgments → Error-biased document sampling → LLM generates relevance labels at scale → Prompt optimization with DSPy.

Related back office ops cases