Ecommerce ops · Production

Flipkart research: semi-supervised DPO fine-tuning of compact VLMs improves product attribute prediction accuracy from 75.1% to 85.7%

The problem

Manually labeling product attributes at e-commerce catalog scale is expensive and error-prone, large VLM APIs cost too much for production use, and a large pool of unlabeled product images remains underutilized.

First attempt

A Self-Learning approach that retrains the model on its own high-confidence predictions caused accuracy to drop in smaller VLMs due to model collapse, where the model reinforces its own biases.

Workflow diagram · grounded in source
1
Initial supervised fine-tuning
ai_action
“We begin by fine tuning a pre-trained VLM on a small, curated dataset where the product attributes have been labeled using a large multimodal model (like Gemini/GPT). To keep the training efficient, we use Parameter-Efficient Fine-Tuning…”
2
Pseudo-label generation via self-consistency
ai_action
“For each unlabelled product, the model is prompted to generate multiple "reasoning-and-answer" chains. We then use a technique called self-consistency to determine the most likely correct answer. The answer that appears most frequently i…”
3
DPO preference optimization
ai_action
“the model is fine-tuned using DPO. This technique teaches the model to favour the "preferred" reasoning chains over the "dis-preferred" ones.”
4
Iterative self-improvement cycle
feedback_loop
“This process is iterative, meaning the model continuously improves by generating better pseudo-labels in each cycle.”
Reported outcome

DPO-based semi-supervised fine-tuning improved accuracy from 75.1% to 85.7% on the tested compact VLM, and increasing unlabeled data volume steadily raised performance further.

Reported metrics
Qwen2.5-VL-3B-Instruct accuracy (baseline supervised)75.1%
Qwen2.5-VL-3B-Instruct accuracy (post-DPO)85.7%
DPO improvement over supervised baselinesignificant improvement over the initial supervised model
Reported stack
VLMsDPOPEFTQwen2.5-VL-3B-InstructGeminiGPT
Source
https://blog.flipkart.tech/the-future-of-e-commerce-how-ai-is-learning-to-describe-products-with-less-data-8dfbf05f83a1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

DPO-based semi-supervised fine-tuning improved accuracy from 75.1% to 85.7% on the tested compact VLM, and increasing unlabeled data volume steadily raised performance further.

What tools did this team use?

VLMs, DPO, PEFT, Qwen2.5-VL-3B-Instruct, Gemini, GPT.

What results were reported?

Qwen2.5-VL-3B-Instruct accuracy (baseline supervised): 75.1%; Qwen2.5-VL-3B-Instruct accuracy (post-DPO): 85.7%; DPO improvement over supervised baseline: significant improvement over the initial supervised model (source-reported, not independently verified).

What failed first in this deployment?

A Self-Learning approach that retrains the model on its own high-confidence predictions caused accuracy to drop in smaller VLMs due to model collapse, where the model reinforces its own biases.

How is this ecommerce ops AI workflow structured?

Initial supervised fine-tuning → Pseudo-label generation via self-consistency → DPO preference optimization → Iterative self-improvement cycle.

Related ecommerce ops cases