Ecommerce ops · Production

Mercari AI Assist: GPT-4 and GPT-3.5-turbo power seller listing title suggestions in production

The problem

Processing and understanding unstructured user-generated listing text was difficult for Mercari's team, as distilling key information and identifying what made listings sell quickly was complex given the varied writing styles of sellers.

Workflow diagram · grounded in source
1
Domain experts define title criteria
human_review
“defining "what makes a good title" for a Mercari listing. This is accomplished with assistance from other teams that possess diverse domain expertise”
2
GPT-4 distills key title attributes
ai_action
“We then collect existing title data aligned with our criteria and utilize GPT-4 to distill the key attributes of an effective title. These key attributes are subsequently stored in a database.”
3
Seller creates listing
trigger
“from a specific listing as it is created”
4
GPT-3.5-turbo extracts listing attributes
ai_action
“We employ GPT-3.5-turbo to identify key attributes (defined by the previous step) from a specific listing as it is created”
5
Title suggestions delivered to seller
output
“we generate suggestions for refining the listing's title as necessary”
Reported outcome

Mercari shipped Mercari AI Assist with a two-stage LLM pipeline—GPT-4 for offline key-attribute extraction and GPT-3.5-turbo for real-time inference—achieving an optimal balance between quality and cost-efficiency.

Reported metrics
GPT-4 vs GPT-3.5-turbo quality tradeoffGPT-4 outperforms GPT-3.5-turbo in terms of quality, but it incurs greater costs and latency
LLM output format inconsistency at scalenumber of inconsistently formatted responses increased along with the number of requests
Reported stack
GPT-4GPT-3.5-turbo
Source
https://engineering.mercari.com/en/blog/entry/20231219-leveraging-llms-in-production-looking-back-going-forward/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Mercari shipped Mercari AI Assist with a two-stage LLM pipeline—GPT-4 for offline key-attribute extraction and GPT-3.5-turbo for real-time inference—achieving an optimal balance between quality and cost-efficiency.

What tools did this team use?

GPT-4, GPT-3.5-turbo.

What results were reported?

GPT-4 vs GPT-3.5-turbo quality tradeoff: GPT-4 outperforms GPT-3.5-turbo in terms of quality, but it incurs greater costs and latency; LLM output format inconsistency at scale: number of inconsistently formatted responses increased along with the number of requests (source-reported, not independently verified).

How is this ecommerce ops AI workflow structured?

Domain experts define title criteria → GPT-4 distills key title attributes → Seller creates listing → GPT-3.5-turbo extracts listing attributes → Title suggestions delivered to seller.

Related ecommerce ops cases