Data entry ops · Production

Deque uses Labelbox Model Diagnostics and Catalog to improve ML model performance 5%+ and cut labeling spend by over 50%

The problem

Deque's ML team spent about 10 minutes annotating each webpage screen and relied on disparate open-source annotation tools hacked together with Jupyter Notebooks and Google Sheets to manually calculate model metrics, making dataset curation and error diagnosis a disjointed, labor-intensive process.

First attempt

Before Labelbox, the team had to manually visualize model predictions and calculate metrics themselves, and data selection for poorly-performing classes required tedious, untargeted collection.

Workflow diagram · grounded in source
1
Webpage screen annotation
Trigger
It takes an average of about 10 minutes to fully annotate a single webpage screen.
▾ source quote
“It takes an average of about 10 minutes to fully annotate a single webpage screen”
2
Model Diagnostics noise detection
Ai action
Model Diagnostics in Labelbox is used to target model weaknesses and detect noise issues in the dataset.
▾ source quote
“leveraged Model Diagnostics and Catalog to target their model's weaknesses and detect noise issues in their datasets”
3
Low-trust data filtering
Validation
About one-third of data points considered less trustworthy are filtered out to improve model performance.
▾ source quote
“we were able to filter out about one-third of data points we considered less trustworthy”
4
Catalog targeted data prioritization
Ai action
Catalog uses embeddings for unsupervised classification to target and prioritize the right data for collection and batch creation.
▾ source quote
“Embeddings allow us to do unsupervised classification of models and select a lot of models. It's just easier to create batches and sample around that”
5
Re-labeling and model iteration
Feedback loop
The team re-labels targeted data points and adds them back to the training set, observing model performance rise further with each round.
▾ source quote
“We re-labeled some data and we saw the performance went up again after we added the re-labeled points”
Reported outcome

Using Labelbox Model Diagnostics and Catalog, Deque filtered out about one-third of less trustworthy data points, raised model performance by 5%+, and reduced annual labeling spend and needs by over 50%.

Reported metrics
Annotation time per screenabout 10 minutes
Data points filtered out as less trustworthyabout one-third
Model performance improvement5%+
Annual labeling spend and needs reductionover 50%
Show all 8 reported metrics
annotation time per screenabout 10 minutes
data points filtered out as less trustworthyabout one-third
model performance improvement5%+
annual labeling spend and needs reductionover 50%
checkbox detection accuracy improvement47% accuracy to 75% accuracy
presentational tables detection accuracy improvement66% accuracy to 79% accuracy
radio button detection accuracy improvement37.9% accuracy to 74% accuracy
labeling effort reductionhalf the time and with half the effort
Reported stack
LabelboxModel DiagnosticsCatalogJupyter NotebooksGoogle Sheets
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://labelbox.com/customers/deque
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Using Labelbox Model Diagnostics and Catalog, Deque filtered out about one-third of less trustworthy data points, raised model performance by 5%+, and reduced annual labeling spend and needs by over 50%.

What tools did this team use?

Labelbox, Model Diagnostics, Catalog, Jupyter Notebooks, Google Sheets.

What results were reported?

Annotation time per screen: about 10 minutes; Data points filtered out as less trustworthy: about one-third; Model performance improvement: 5%+; Annual labeling spend and needs reduction: over 50% (source-reported, not independently verified).

What failed first in this deployment?

Before Labelbox, the team had to manually visualize model predictions and calculate metrics themselves, and data selection for poorly-performing classes required tedious, untargeted collection.

How is this data entry ops AI workflow structured?

Webpage screen annotation → Model Diagnostics noise detection → Low-trust data filtering → Catalog targeted data prioritization → Re-labeling and model iteration.

Related data entry ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Data entry ops cases →