Data entry ops · Production

Deque uses Labelbox Model Diagnostics and Catalog to improve ML model performance 5%+ and cut labeling spend by over 50%

The problem

Deque's ML team spent about 10 minutes annotating each webpage screen and relied on disparate open-source annotation tools hacked together with Jupyter Notebooks and Google Sheets to manually calculate model metrics, making dataset curation and error diagnosis a disjointed, labor-intensive process.

First attempt

Before Labelbox, the team had to manually visualize model predictions and calculate metrics themselves, and data selection for poorly-performing classes required tedious, untargeted collection.

Workflow diagram · grounded in source
1
Webpage screen annotation
trigger
“It takes an average of about 10 minutes to fully annotate a single webpage screen”
2
Model Diagnostics noise detection
ai_action
“leveraged Model Diagnostics and Catalog to target their model's weaknesses and detect noise issues in their datasets”
3
Low-trust data filtering
validation
“we were able to filter out about one-third of data points we considered less trustworthy”
4
Catalog targeted data prioritization
ai_action
“Embeddings allow us to do unsupervised classification of models and select a lot of models. It's just easier to create batches and sample around that”
5
Re-labeling and model iteration
feedback_loop
“We re-labeled some data and we saw the performance went up again after we added the re-labeled points”
Reported outcome

Using Labelbox Model Diagnostics and Catalog, Deque filtered out about one-third of less trustworthy data points, raised model performance by 5%+, and reduced annual labeling spend and needs by over 50%.

Reported metrics
Annotation time per screenabout 10 minutes
Data points filtered out as less trustworthyabout one-third
Model performance improvement5%+
Annual labeling spend and needs reductionover 50%
Show all 8 reported metrics
annotation time per screenabout 10 minutes
data points filtered out as less trustworthyabout one-third
model performance improvement5%+
annual labeling spend and needs reductionover 50%
checkbox detection accuracy improvement47% accuracy to 75% accuracy
presentational tables detection accuracy improvement66% accuracy to 79% accuracy
radio button detection accuracy improvement37.9% accuracy to 74% accuracy
labeling effort reductionhalf the time and with half the effort
Reported stack
LabelboxModel DiagnosticsCatalogJupyter NotebooksGoogle Sheets
Source
https://labelbox.com/customers/deque
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Using Labelbox Model Diagnostics and Catalog, Deque filtered out about one-third of less trustworthy data points, raised model performance by 5%+, and reduced annual labeling spend and needs by over 50%.

What tools did this team use?

Labelbox, Model Diagnostics, Catalog, Jupyter Notebooks, Google Sheets.

What results were reported?

Annotation time per screen: about 10 minutes; Data points filtered out as less trustworthy: about one-third; Model performance improvement: 5%+; Annual labeling spend and needs reduction: over 50% (source-reported, not independently verified).

What failed first in this deployment?

Before Labelbox, the team had to manually visualize model predictions and calculate metrics themselves, and data selection for poorly-performing classes required tedious, untargeted collection.

How is this data entry ops AI workflow structured?

Webpage screen annotation → Model Diagnostics noise detection → Low-trust data filtering → Catalog targeted data prioritization → Re-labeling and model iteration.

Related data entry ops cases