Ecommerce ops · Production

One Prompt To Rule Them All: LLMs For Opinion Summary Evaluation at Flipkart

The problem

Traditional automatic metrics like ROUGE fail to provide comprehensive assessment of opinion summaries and show poor alignment with human judgment, leaving e-commerce teams without a reliable way to evaluate AI-generated review summaries.

First attempt

Reference-based metrics ROUGE and BERTSCORE showed very poor and sometimes negative correlation with human ratings of summary quality, confirming they are inadequate for assessing modern generative model outputs.

Workflow diagram · grounded in source
1
Customer browsing triggers review overload
trigger
“Imagine a customer on Flipkart browsing for a new smartphone, confronted with hundreds of user reviews”
2
AI generates opinion summary
ai_action
“using AI to analyze all customer reviews and generate a brief, balanced summary”
3
OP-I-PROMPT structures evaluation prompt
validation
“OP-I-PROMPT uses a modular design. It uses a fixed skeleton prompt containing the Task Description, Evaluation Criteria (a 1–5 scale), and Evaluation Steps. To test a different dimension, you simply swap out a single block of text: the M…”
4
LLM chain-of-thought scoring
ai_action
“The prompt explicitly instructs the model to perform a step-by-step analysis before assigning a score. This forces the model to justify its reasoning, leading to evaluations that correlate much more closely with human judgment.”
5
Summaries drive business outcomes
output
“high-quality summaries can drive key business metrics like increased conversion rates and reduced product returns”
Reported outcome

OP-I-PROMPT achieved a Spearman correlation of 0.70 with human judgments, outperforming G-EVAL on open-source models, while the Flipkart use case demonstrated that high-quality summaries can drive increased conversion rates and reduced product returns.

Reported metrics
Spearman correlation with human judgments0.70
Expert annotations in SUMMEVAL-OP dataset2,912
Conversion ratesincreased conversion rates
Product return ratesreduced product return rates
Show all 5 reported metrics
Spearman correlation with human judgments0.70
Expert annotations in SUMMEVAL-OP dataset2,912
Conversion ratesincreased conversion rates
Product return ratesreduced product return rates
Customer satisfactionimprove customer satisfaction
Reported stack
OP-I-PROMPTSUMMEVAL-OPROUGEBERTSCOREChatGPT-3.5GPT-4Solar-10.7B
Source
https://blog.flipkart.tech/one-prompt-to-rule-them-all-llms-for-opinion-summary-evaluation-d5dd4eb6f225
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

OP-I-PROMPT achieved a Spearman correlation of 0.70 with human judgments, outperforming G-EVAL on open-source models, while the Flipkart use case demonstrated that high-quality summaries can drive increased conversion…

What tools did this team use?

OP-I-PROMPT, SUMMEVAL-OP, ROUGE, BERTSCORE, ChatGPT-3.5, GPT-4, Solar-10.7B.

What results were reported?

Spearman correlation with human judgments: 0.70; Expert annotations in SUMMEVAL-OP dataset: 2,912; Conversion rates: increased conversion rates; Product return rates: reduced product return rates (source-reported, not independently verified).

What failed first in this deployment?

Reference-based metrics ROUGE and BERTSCORE showed very poor and sometimes negative correlation with human ratings of summary quality, confirming they are inadequate for assessing modern generative model outputs.

How is this ecommerce ops AI workflow structured?

Customer browsing triggers review overload → AI generates opinion summary → OP-I-PROMPT structures evaluation prompt → LLM chain-of-thought scoring → Summaries drive business outcomes.

Related ecommerce ops cases