Quality assurance · Production

How a leading AI lab fuels agentic development with frontier data via Labelbox

The problem

A leading AI lab needed a rigorous method to evaluate agentic models beyond basic instruction-following, specifically assessing how models plan, reason, and adapt across complex multi-step tool-use tasks in scenarios mirroring real-world ambiguity.

Workflow diagram · grounded in source
1
Lab identifies evaluation need
trigger
“A leading lab needed a robust method to test their models in scenarios mirroring real-world ambiguity, such as booking intricate travel arrangements or managing nuanced retail returns”
2
Design domain environments
integration
“Labelbox created multiple domain environments, each with its own database and API interfaces, supporting 15-20 API functions”
3
Tier 1: Define API schemas
human_review
“One highly specialized group of trainers focused on the architectural foundation: building the interfaces, meticulously defining tool schemas, and ensuring the technical accuracy of each API function”
4
Tier 2: Create task prompts
human_review
“another group, equally expert but with a focus on real-world applicability, invented the realistic task prompts and mapped out the ground truth action plans”
5
Model executes multi-step tasks
ai_action
“testing the model's ability to interpret natural language, plan tool calls, adapt to responses, and reach a verifiable end state”
6
Score difficulty and verify outcomes
validation
“Each task was scored for difficulty and verified through structured outputs”
7
Surface reasoning gaps
feedback_loop
“helping the team surface key gaps in reasoning and execution”
Reported outcome

Labelbox delivered a benchmark comprising 25 interfaces, over 250 API functions, and more than 1,000 tool-use tasks, giving the AI lab an objective mechanism to pressure-test agentic model performance, surface key gaps in reasoning and execution, and accelerate product development and model iteration.

Reported metrics
API interfaces in benchmark25
API functions in benchmarkover 250
Tool-use tasks in benchmarkmore than 1,000
Product and model iteration speedaccelerating both product development and model iteration
Reported stack
LabelboxPython
Source
https://labelbox.com/customers/agentic-development-customer-story
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Labelbox delivered a benchmark comprising 25 interfaces, over 250 API functions, and more than 1,000 tool-use tasks, giving the AI lab an objective mechanism to pressure-test agentic model performance, surface key gap…

What tools did this team use?

Labelbox, Python.

What results were reported?

API interfaces in benchmark: 25; API functions in benchmark: over 250; Tool-use tasks in benchmark: more than 1,000; Product and model iteration speed: accelerating both product development and model iteration (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Lab identifies evaluation need → Design domain environments → Tier 1: Define API schemas → Tier 2: Create task prompts → Model executes multi-step tasks → Score difficulty and verify outcomes → Surface reasoning gaps.

Related quality assurance cases