Quality assurance · Production

Galileo surfaces and fixes 6.5% of ML dataset errors in 10 minutes with 7.24% model performance improvement

The problem

ML datasets in production have critical quality issues — including empty samples, garbage samples, labeling errors, and class overlap — but most modern tools focus on changing model configurations or collecting more data rather than addressing root data quality flaws.

Workflow diagram · grounded in source
1
Dataset loaded into Galileo
trigger
“we analyze various benchmark datasets in academia and other industries using Galileo, and surface crucial errors and ambiguities within minutes”
2
Embeddings visualization finds empty samples
ai_action
“Galileo's embeddings visualization quickly surfaces an island far from the rest of the data. By selecting the samples in the cluster and taking a further look at them in Galileo's dataframe view, we see over 200 empty samples! Additional…”
3
DEP score flags labeling errors
ai_action
“Using Galileo's Data Error Potential (DEP) score as a guide, we uncover samples that are confusing to the model, either because of human labeling errors or semantic overlap between classes”
4
Class-level metrics reveal class overlap
ai_action
“By looking at class-level metrics in Galileo, we observe that the talk.religion.misc class is particularly low performing. Furthermore, this label has the highest DEP score and lowest class distribution, with only 323 samples in the trai…”
5
Human annotator inspects samples
human_review
“we additionally inspect selected samples with a human annotator”
6
None label added and dataset fixed
output
“we propose adding a new "None" label to the dataset. In this way, the model can become robust towards such garbage samples through learning to identify them as a separate class”
Reported outcome

Galileo identified and fixed 1163 dataset errors (6.5% of 18,000 samples) within 10 minutes, producing a 7.24% overall model performance improvement.

Reported metrics
Dataset errors fixed (count)1163
Dataset errors fixed (percentage of total)6.5%
Time to fix dataset10 minutes
Overall model performance improvement7.24%
Show all 13 reported metrics
dataset errors fixed (count)1163
dataset errors fixed (percentage of total)6.5%
time to fix dataset10 minutes
overall model performance improvement7.24%
malformed samples identified6.7%
short samples in dataset~7%
empty samples foundover 200
labeling error samples uncovered290
talk.religion.misc samples flagged as hardover 50%
dataset share affected by class overlap2.8%
baseline model F1 on test split0.69
talk.religion.misc training samples (count)323
time without Galileo for same taskseveral days of expensive debugging and reannotation effort
Reported stack
GalileoDistil-BERT
Source
https://mlops.community/blog/improving-your-ml-datasets-with-galileo
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Galileo identified and fixed 1163 dataset errors (6.5% of 18,000 samples) within 10 minutes, producing a 7.24% overall model performance improvement.

What tools did this team use?

Galileo, Distil-BERT.

What results were reported?

Dataset errors fixed (count): 1163; Dataset errors fixed (percentage of total): 6.5%; Time to fix dataset: 10 minutes; Overall model performance improvement: 7.24% (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Dataset loaded into Galileo → Embeddings visualization finds empty samples → DEP score flags labeling errors → Class-level metrics reveal class overlap → Human annotator inspects samples → None label added and dataset fixed.

Related quality assurance cases