Quality assurance · Production

Dropbox uses DSPy to optimize Dash relevance judge: 45% NMSE reduction and 97% fewer malformed outputs

The problem

Dropbox's relevance judge for Dash was built on an expensive state-of-the-art model that could not be scaled cost-effectively, and manually tuned prompts did not transfer cleanly to cheaper models, causing quality to drop and requiring weeks of manual iteration per model swap.

First attempt

Manual prompt engineering for the relevance judge plateaued in quality and broke when transferring prompts between models. With gemma-3-12b, more than 40 percent of responses were malformed JSON, making the baseline operationally unusable.

Workflow diagram · grounded in source

Query-document pair submitted

trigger

“given a query and a document, it assigns a relevance score from 1 to 5, where 5 indicates a perfect match and 1 indicates no meaningful connection to the query and user intent”

Human annotators score pairs

human_review

“humans are shown a query and a candidate document and asked to rate its relevance on that same 1–5 scale. They also provide a short explanation describing why they chose that score”

NMSE alignment measurement

validation

“We then measure how far the model's ratings deviate from the human ratings using normalized mean squared error (NMSE)”

GEPA structured feedback generation

ai_action

“GEPA generates structured feedback for each example where the model disagrees with a human annotator. In our case, we combined the size and direction of the gap with the human explanation and the model's reasoning, producing concrete sig…”

Iterative prompt revision loop

feedback_loop

“The prompt is evaluated, its failure modes are surfaced in plain language, the prompt is revised, and the cycle repeats—all while directly optimizing against the human-alignment metric defined earlier”

Optimized judge deployed to pipelines

output

“relevance scoring is a core capability that shapes ranking, training data generation, and offline simulation”

Reported outcome

DSPy-optimized prompts reduced NMSE by 45 percent (from 8.83 to 4.86) when adapting to gpt-oss-120b, cut model adaptation time from one to two weeks down to one to two days, enabled labeling 10 to 100 times more data at the same cost, and reduced malformed JSON outputs by more than 97 percent when adapting to gemma-3-12b.

Reported metrics

NMSE reduction (gpt-oss-120b adaptation)45 percent

NMSE before DSPy (gpt-oss-120b)8.83

NMSE after DSPy (gpt-oss-120b)4.86

Model adaptation time before DSPyone to two weeks

Show all 10 reported metrics

NMSE reduction (gpt-oss-120b adaptation)45 percent

NMSE before DSPy (gpt-oss-120b)8.83

NMSE after DSPy (gpt-oss-120b)4.86

Model adaptation time before DSPyone to two weeks

Model adaptation time after DSPyone to two days

Data labeling capacity increase10–100 times more data at the same cost

Malformed JSON output reduction (gemma-3-12b)dropped by more than 97 percent

Malformed JSON baseline rate (gemma-3-12b)more than 40 percent

NMSE baseline gemma-3-12b (original prompt)46.88

NMSE after DSPy MIPROv2 (gemma-3-12b)17.26

Reported stack

DSPyGEPAMIPROv2gpt-oss-120bgemma-3-12bo3Dropbox Dash

Source

https://dropbox.tech/machine-learning/optimizing-dropbox-dash-relevance-judge-with-dspy

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

DSPy-optimized prompts reduced NMSE by 45 percent (from 8.83 to 4.86) when adapting to gpt-oss-120b, cut model adaptation time from one to two weeks down to one to two days, enabled labeling 10 to 100 times more data…

What tools did this team use?

DSPy, GEPA, MIPROv2, gpt-oss-120b, gemma-3-12b, o3, Dropbox Dash.

What results were reported?

NMSE reduction (gpt-oss-120b adaptation): 45 percent; NMSE before DSPy (gpt-oss-120b): 8.83; NMSE after DSPy (gpt-oss-120b): 4.86; Model adaptation time before DSPy: one to two weeks (source-reported, not independently verified).

What failed first in this deployment?

Manual prompt engineering for the relevance judge plateaued in quality and broke when transferring prompts between models.

How is this quality assurance AI workflow structured?

Query-document pair submitted → Human annotators score pairs → NMSE alignment measurement → GEPA structured feedback generation → Iterative prompt revision loop → Optimized judge deployed to pipelines.