Quality assurance · Production

Deque uses Labelbox Model Diagnostics and Catalog to improve accessibility ML model performance by 5%+ and cut labeling spend by over 50%

The problem

The Deque team spent approximately 10 minutes annotating each webpage screen for accessibility compliance and managed thousands of web and mobile datasets using disparate open-source annotation tools combined with ad-hoc Jupyter Notebooks and Google Sheets, making model diagnostics and data collection a manual, disjointed process.

First attempt

Before Labelbox, Deque's team had to manually visualize predictions and calculate all metrics themselves, and data collection was a scattershot process that would have required roughly twice as much data and effort for equivalent model improvement.

Workflow diagram · grounded in source
1
Model Diagnostics identifies weaknesses
ai_action
“Before using Model Diagnostics in Labelbox to target the model's weaknesses, we had to visualize the predictions on our own and everything was much more manual”
2
Noise-based data filtering
validation
“we detected some noise issues in our dataset and thanks to Model Diagnostics, we were able to filter out about one-third of data points we considered less trustworthy”
3
Catalog prioritizes target data
ai_action
“Embeddings allow us to do unsupervised classification of models and select a lot of models. It's just easier to create batches and sample around that”
4
Re-labeling improves performance
output
“We re-labeled some data and we saw the performance went up again after we added the re-labeled points”
Reported outcome

By filtering out one-third of less trustworthy data points and targeting data collection via Model Diagnostics and Catalog, Deque improved overall model performance by 5%+ and reduced annual labeling spend and needs by over 50%.

Reported metrics
Overall model performance improvement5%+
Annual labeling spend and needsover 50%
Less trustworthy data points filtered1/3 of data points considered less trustworthy
Checkbox detection accuracy improvement47% to 75%
Show all 8 reported metrics
overall model performance improvement5%+
annual labeling spend and needsover 50%
less trustworthy data points filtered1/3 of data points considered less trustworthy
checkbox detection accuracy improvement47% to 75%
presentational table detection accuracy improvement66% to 79%
radio button detection accuracy improvement37.9% to 74%
annotation time per webpage screenaverage of about 10 minutes
labeling effort vs scattershot approachroughly labeling twice as much data and with twice as much effort
Reported stack
LabelboxModel DiagnosticsCatalog
Source
https://labelbox.com/customers/deque/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By filtering out one-third of less trustworthy data points and targeting data collection via Model Diagnostics and Catalog, Deque improved overall model performance by 5%+ and reduced annual labeling spend and needs b…

What tools did this team use?

Labelbox, Model Diagnostics, Catalog.

What results were reported?

Overall model performance improvement: 5%+; Annual labeling spend and needs: over 50%; Less trustworthy data points filtered: 1/3 of data points considered less trustworthy; Checkbox detection accuracy improvement: 47% to 75% (source-reported, not independently verified).

What failed first in this deployment?

Before Labelbox, Deque's team had to manually visualize predictions and calculate all metrics themselves, and data collection was a scattershot process that would have required roughly twice as much data and effort fo…

How is this quality assurance AI workflow structured?

Model Diagnostics identifies weaknesses → Noise-based data filtering → Catalog prioritizes target data → Re-labeling improves performance.

Related quality assurance cases