Data entry ops · Production

Leading EdTech enterprise scales ML training data annotation with Labelbox

The problem

Rapid pandemic growth created demand for hundreds of thousands of annotated text data items to train ML models, but the company's existing tools — Amazon SageMaker GroundTruth, Prodigy, and an in-house labeling tool — could not scale to meet that demand.

First attempt

Previous annotation services including Amazon SageMaker GroundTruth and Prodigy, as well as an in-house tool, lacked transparency and offered no ability to revisit submitted labels, fix errors, or track labeler productivity — making them feel like a black box.

Workflow diagram · grounded in source
1
Student screenshot responses
trigger
“Students answering questions from their software were typically done in the form of screenshot images”
2
OCR to text conversion
ai_action
“needing to convert OCR data back into text data so that experts can more effectively annotate them”
3
In-house OCR accuracy comparison
validation
“The enterprise also built an in-house comparison system within Labelbox for measuring the accuracy of different OCR labels”
4
Expert text annotation
human_review
“start the process of labeling hundreds of thousands of text data for their ML models”
5
QA with consensus tools
validation
“quality assurance using consensus tools”
6
Productivity and quality tracking
output
“Labelbox provides the ability to count the number of labels done, revisit submitted labels, fix errors, run a full quality assurance pipeline and manage labeler productivity”
7
AI-assisted question-answering
ai_action
“AI and ML is also now able to make the complex question-answering process smoother and also help speed up the answering process for experts. The streamlined workflow helps populate data fields faster and make better question/answer recom…”
Reported outcome

The company rapidly delivered hundreds of thousands of annotations in record speed while gaining full visibility into labeler productivity and training data quality; AI and ML also made the question-answering process smoother and helped speed up expert workflows.

Reported metrics
Annotations deliveredhundreds of thousands of annotations
Annotation delivery speedrecord speed
Student learning and outcomesboost student learning and outcomes
Reported stack
LabelboxWorkforceOCRAmazon SageMaker GroundTruthProdigy
Source
https://labelbox.com/customers/ner-edtech/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The company rapidly delivered hundreds of thousands of annotations in record speed while gaining full visibility into labeler productivity and training data quality; AI and ML also made the question-answering process…

What tools did this team use?

Labelbox, Workforce, OCR, Amazon SageMaker GroundTruth, Prodigy.

What results were reported?

Annotations delivered: hundreds of thousands of annotations; Annotation delivery speed: record speed; Student learning and outcomes: boost student learning and outcomes (source-reported, not independently verified).

What failed first in this deployment?

Previous annotation services including Amazon SageMaker GroundTruth and Prodigy, as well as an in-house tool, lacked transparency and offered no ability to revisit submitted labels, fix errors, or track labeler produc…

How is this data entry ops AI workflow structured?

Student screenshot responses → OCR to text conversion → In-house OCR accuracy comparison → Expert text annotation → QA with consensus tools → Productivity and quality tracking → AI-assisted question-answering.

Related data entry ops cases