Quality assurance · Production

Dropbox's evaluation-first blueprint for Dash: LLM judges, automated gates, and continuous improvement at scale

The problem

Building Dropbox Dash exposed a fundamental challenge: LLM pipelines are probabilistic chains where a single prompt tweak can silently break production quality, and early ad-hoc evaluation gave no reliable way to catch regressions before they reached users.

First attempt

Traditional NLP metrics like BLEU and ROUGE failed to detect hallucinations and missed citations; spreadsheet-based experiment tracking broke down under real experimentation; and unstructured prompt changes caused surprise regressions that slipped into production.

Workflow diagram · grounded in source
1
PR triggers automated evaluation
trigger
“Every pull request kicked off about 150 canonical queries, which were judged automatically and returned results in under 10 minutes”
2
LLM judge scores outputs
ai_action
“Does the answer directly address the query? Are all factual claims supported by the provided context? Is the answer clear, well-formatted, and consistent in voice?”
3
Gate check blocks regressions
validation
“If performance slipped below the thresholds, the change didn't move forward”
4
Human spot-audit on release
human_review
“human engineers manually reviewed 5–10% of the regression suite. Any discrepancies were logged and traced back to either prompt bugs or model hallucinations, and recurring issues triggered prompt rewrites or more fine-grained scoring”
5
Live traffic continuous scoring
feedback_loop
“we continuously sampled live production traffic and scored it with the same metrics and logic as our offline suites”
6
Failure mining updates datasets
feedback_loop
“By mining low-rated traces from live traffic, we uncovered failure patterns that synthetic datasets often missed”
Reported outcome

Dropbox established an evaluation-first engineering culture where automated gates catch regressions at the pull-request level before code can merge, and live production traffic is continuously scored to detect silent degradations.

Reported metrics
Source F1 quality threshold0.85
P95 latency threshold5s
Canonical queries per PR checkabout 150
PR evaluation turnaround timeunder ten minutes
Show all 5 reported metrics
Source F1 quality threshold0.85
p95 latency threshold5s
Canonical queries per PR checkabout 150
PR evaluation turnaround timeunder ten minutes
Human regression suite review rate5–10%
Reported stack
BraintrustKubeflowGitHub ActionNatural QuestionsMS MARCOMuSiQue
Source
https://dropbox.tech/machine-learning/practical-blueprint-evaluating-conversational-ai-at-scale-dash
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Dropbox established an evaluation-first engineering culture where automated gates catch regressions at the pull-request level before code can merge, and live production traffic is continuously scored to detect silent…

What tools did this team use?

Braintrust, Kubeflow, GitHub Action, Natural Questions, MS MARCO, MuSiQue.

What results were reported?

Source F1 quality threshold: 0.85; P95 latency threshold: 5s; Canonical queries per PR check: about 150; PR evaluation turnaround time: under ten minutes (source-reported, not independently verified).

What failed first in this deployment?

Traditional NLP metrics like BLEU and ROUGE failed to detect hallucinations and missed citations; spreadsheet-based experiment tracking broke down under real experimentation; and unstructured prompt changes caused sur…

How is this quality assurance AI workflow structured?

PR triggers automated evaluation → LLM judge scores outputs → Gate check blocks regressions → Human spot-audit on release → Live traffic continuous scoring → Failure mining updates datasets.

Related quality assurance cases