Quality assurance · Production

Uber AI Solutions' Requirement Adherence: LLMs reduce data labeling audit requirements by 80%

The problem

Data labeling workflows relying on post-labeling checks and interhuman agreement caused mislabeled data to be sent back to experts for rework, increasing time and costs and creating a bad client experience. The diverse nature of client requirements also made creating custom quality solutions per project unscalable.

First attempt

A single LLM call to enforce all requirements at once led to hallucinations and missed enforcements, making a single-call approach unreliable for quality checking.

Workflow diagram · grounded in source

SOP document intake

trigger

“This document, which includes all client requirements along with other information, is either provided by the client or compiled collaboratively by an Uber AI Solutions Program Manager and the client.”

SOP-to-markdown conversion and rule extraction

ai_action

“We've designed our rule extraction logic using LLMs with a reflection capability. This allows the LLM to analyze the SOP text, generate a structured JSON output, check its work, and make changes if necessary.”

Human review adds manual rules

human_review

“We also allow for manual rules to be added at this stage to cover any gaps in the rules the LLM extracted or to add new requirements not included in the SOP document. We found that this human-in-the-loop step is useful to ensure high-qua…”

Rule format validation and deduplication

validation

“we use another LLM call to ensure that the manual rules follow the required format and there's no overlap between them and the auto-extracted rules”

Rule complexity routing

routing

“Different rule types leverage different LLM strengths. Formatting checks are done using code, so an LLM isn't involved. Deterministic checks use a non-reasoning LLM for speed and efficiency. Subjective and complex subjective checks use p…”

Real-time parallel validation in uLabel

ai_action

“Instead of a single, monolithic LLM call for all checks, we now perform a single validation call per rule. These calls can run in parallel, significantly speeding up the feedback loop for labelers. We rely on prefix caching to reduce the…”

Suggestions and grammar check output

output

“we also provide suggestions to the expert on the changes required to fix the quality issues. Alongside these validations, we also run spelling and grammar checks to assist the expert.”

Feedback collection for prompt optimization

feedback_loop

“we're building mechanisms to collect feedback on validations. This data is crucial for optimizing our LLM prompts over time, with a long-term goal of auto-optimizing them based on real-world performance.”

Reported outcome

In-tool validation produced a substantial enhancement in annotation quality and an 80% reduction in audits required, helping meet timelines and reduce costs.
The framework became a standard, widely adopted step across the entire client base.

Reported metrics

Reduction in audits required80%

In-tool annotation qualitysubstantial enhancement

Reported stack

LLMsuLabel

Source

https://www.uber.com/en-IN/blog/requirement-adherence-boosting-data-labeling-quality-using-llms/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

In-tool validation produced a substantial enhancement in annotation quality and an 80% reduction in audits required, helping meet timelines and reduce costs.

What tools did this team use?

LLMs, uLabel.

What results were reported?

Reduction in audits required: 80%; In-tool annotation quality: substantial enhancement (source-reported, not independently verified).

What failed first in this deployment?

A single LLM call to enforce all requirements at once led to hallucinations and missed enforcements, making a single-call approach unreliable for quality checking.

How is this quality assurance AI workflow structured?

SOP document intake → SOP-to-markdown conversion and rule extraction → Human review adds manual rules → Rule format validation and deduplication → Rule complexity routing → Real-time parallel validation in uLabel → Suggestions and grammar check output → Feedback collection for prompt optimization.

In-tool validation produced a substantial enhancement in annotation quality and an 80% reduction in audits required, helping meet timelines and reduce costs.The framework became a standard, widely adopted step across the entire client base.