Data entry ops · Production

Kantar Worldpanel uses Databricks and GPT-4 to generate 120,000 training pairs at 94% accuracy

The problem

Kantar Worldpanel's legacy systems were inflexible, resource-intensive, and required an outdated programming skillset, blocking their ability to experiment with modern AI/ML models and scale their data insights business.

Workflow diagram · grounded in source
1
Receipt-to-barcode matching need
trigger
“Pairing receipt line descriptions and product barcode names is just one of the use cases in which we're trying to leverage GenAI”
2
Multi-LLM experimentation
ai_action
“We've experimented with Llama, Mistral, GPT-4 and GPT-3.5, all within the Databricks Platform. Ultimately, GPT-4 provided better answers, with an accuracy of 94%”
3
Automated training data generation
ai_action
“automatically generated a training dataset of about 120,000 pairs of receipt descriptions and barcode names with an accuracy of 94% in just a couple of hours”
4
Smaller model fine-tuning
ai_action
“we can use it to generate training data to fine-tune a smaller model and serve it in our production pipeline. Smaller models are not only more cost-effective but more performant”
Reported outcome

Kantar Worldpanel automatically generated a training dataset of about 120,000 pairs of receipt descriptions and barcode names at 94% accuracy in a couple of hours, freeing manual coding teams and engineering resources for higher-value work.

Reported metrics
model accuracy (GPT-4 and generated training data)94%
Training dataset size generated120,000 pairs
Time to generate training datasetin just a couple of hours
Resource utilizationsignificant resource optimization
Show all 5 reported metrics
model accuracy (GPT-4 and generated training data)94%
training dataset size generated120,000 pairs
time to generate training datasetin just a couple of hours
resource utilizationsignificant resource optimization
smaller model cost and performancemore cost-effective but more performant
Reported stack
DatabricksMLflowMosaic AI Vector SearchUnity CatalogDatabricks MarketplaceLlamaMistralGPT-4GPT-3.5
Source
https://www.databricks.com/customers/kantar-genai
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Kantar Worldpanel automatically generated a training dataset of about 120,000 pairs of receipt descriptions and barcode names at 94% accuracy in a couple of hours, freeing manual coding teams and engineering resources…

What tools did this team use?

Databricks, MLflow, Mosaic AI Vector Search, Unity Catalog, Databricks Marketplace, Llama, Mistral, GPT-4, GPT-3.5.

What results were reported?

model accuracy (GPT-4 and generated training data): 94%; Training dataset size generated: 120,000 pairs; Time to generate training dataset: in just a couple of hours; Resource utilization: significant resource optimization (source-reported, not independently verified).

How is this data entry ops AI workflow structured?

Receipt-to-barcode matching need → Multi-LLM experimentation → Automated training data generation → Smaller model fine-tuning.

Related data entry ops cases