Quality assurance · Production

Labelbox assembles 150 STEM experts to evaluate and improve a leading AI lab's multimodal LLM

The problem

A leading AI lab needed a reliable team of qualified STEM experts to evaluate their LLM on K-12 domain-specific questions and generate multimodal training data, but faced difficulty sourcing specialists with deep technical expertise across biology, physics, engineering, and related fields.

Workflow diagram · grounded in source
1
AI lab identifies evaluation need
trigger
“A leading AI lab sought to identify areas within K-12 STEM education where their large language model (LLM) struggled to generate accurate responses”
2
Source and vet STEM experts
integration
“Labelbox carefully vetted hundreds of STEM experts in fields like engineering, math, and physics, ultimately selecting 150 highly qualified professionals with PhDs and Master's degrees”
3
Create multimodal STEM prompts
human_review
“The AI trainer was asked to generate complex prompts that include images and text (multimodal) that covered a wide variety of STEM fields from all grade levels. Prompts were adjusted until they pushed the limits of the LLM”
4
Evaluate model and identify winning labels
validation
“Only if the model provided incorrect answers multiple times, was the prompt then considered a 'winning label.'”
5
Create accurate training responses
output
“accurate responses were created to help train the model”
6
Integrate into real-time loss training
feedback_loop
“Labelbox is now a pivotal, fully integrated partner in delivering high-quality, domain-specific STEM data for the lab's real-time loss training workflow”
Reported outcome

Labelbox's team of STEM experts consistently generated unique multimodal reasoning prompts that identified the model's limitations and significantly enhanced its performance on complex STEM questions, with Labelbox now serving as a fully integrated partner in the lab's real-time loss training workflow.

Reported metrics
STEM experts selected150
model performance on complex STEM questionssignificantly enhanced
Expert calibration period24-hour calibration period
Reported stack
LabelboxAlignerrmultimodal chat editor
Source
https://labelbox.com/customers/multimodal-STEM-customer-story
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Labelbox's team of STEM experts consistently generated unique multimodal reasoning prompts that identified the model's limitations and significantly enhanced its performance on complex STEM questions, with Labelbox no…

What tools did this team use?

Labelbox, Alignerr, multimodal chat editor.

What results were reported?

STEM experts selected: 150; model performance on complex STEM questions: significantly enhanced; Expert calibration period: 24-hour calibration period (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

AI lab identifies evaluation need → Source and vet STEM experts → Create multimodal STEM prompts → Evaluate model and identify winning labels → Create accurate training responses → Integrate into real-time loss training.

Related quality assurance cases