Ecommerce ops · Production

Amazon builds COSMO commonsense knowledge graph to improve product recommendation F1 by 60%

The problem

Amazon's product recommendation engine lacked commonsense reasoning to infer contextual product needs, such as that pregnant women searching for shoes might want slip-resistant footwear.

First attempt

LLMs used to generate commonsense relationship hypotheses tend to produce empty or trivial rationales, requiring heuristic filtering and additional validation steps to remove low-quality outputs.

Workflow diagram · grounded in source
1
Customer query submitted
trigger
“If a customer, for instance, submits a query for "shoes for pregnant women", the recommendation engine should be able to deduce that pregnant women might want slip-resistant shoes.”
2
Collect and prune data pairs
integration
“COSMO's knowledge graph construction procedure begins with two types of data: query-purchase pairs, which combine queries with purchases made within a fixed span of time or a fixed number of clicks, and co-purchase pairs, which combine p…”
3
LLM generates candidate relationships
ai_action
“We then feed the data pairs to an LLM and ask it to describe the relationships between the inputs using one of four relationships: usedFor, capableOf, isA, and cause.”
4
Heuristic filtering of candidates
validation
“after the LLM has generated a set of candidate relationships, we apply various heuristics to winnow them down. For instance, if the LLM's answer to our question is semantically too similar to the question itself, we filter out the questi…”
5
Human annotation of candidates
human_review
“we select a representative subset, which we send to human annotators for assessment according to two criteria: plausibility, or whether the posited inferential relationship is reasonable, and typicality, or whether the target product is …”
6
ML classifier scores remaining candidates
ai_action
“Using the annotated data, we train a machine-learning-based classifier that assigns plausibility and typicality scores to the remaining candidates, and we keep only those that exceed some threshold.”
7
Principle extraction re-prompts LLM
feedback_loop
“From those candidates we extract syntactic and semantic relationships that can be encoded as instructions to an LLM, such as "generate explanations for the search-buy behavior in the domain 𝑑 using the capableOf relation". Then we reass…”
8
Knowledge graph assembled from triples
output
“The result is a set of entity-relation-entity triples, such as <co-purchase of camera case and screen protector, capableOf, protecting camera>, from which we assemble a knowledge graph.”
9
COSMO triples integrated into recommendation model
integration
“a second cross-encoder received query-product pairs, along with relevant triples from the COSMO knowledge graph, such as <co-purchase of camera case and screen protector, capable_of, protecting camera>”
Reported outcome

The COSMO-enhanced recommendation model achieved a 60% increase in macro F1 with frozen encoders, and maintained a 28% macro F1 edge and 22% micro F1 edge over the best baseline after fine-tuning.

Reported metrics
macro F1 improvement (frozen encoders)60%
macro F1 edge over baseline (fine-tuned encoders)28%
micro F1 edge over baseline (fine-tuned encoders)22%
Reported stack
COSMOLLMs
Source
https://www.amazon.science/blog/building-commonsense-knowledge-graphs-to-aid-product-recommendation?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The COSMO-enhanced recommendation model achieved a 60% increase in macro F1 with frozen encoders, and maintained a 28% macro F1 edge and 22% micro F1 edge over the best baseline after fine-tuning.

What tools did this team use?

COSMO, LLMs.

What results were reported?

macro F1 improvement (frozen encoders): 60%; macro F1 edge over baseline (fine-tuned encoders): 28%; micro F1 edge over baseline (fine-tuned encoders): 22% (source-reported, not independently verified).

What failed first in this deployment?

LLMs used to generate commonsense relationship hypotheses tend to produce empty or trivial rationales, requiring heuristic filtering and additional validation steps to remove low-quality outputs.

How is this ecommerce ops AI workflow structured?

Customer query submitted → Collect and prune data pairs → LLM generates candidate relationships → Heuristic filtering of candidates → Human annotation of candidates → ML classifier scores remaining candidates → Principle extraction re-prompts LLM → Knowledge graph assembled from triples → COSMO triples integrated into recommendation model.

Related ecommerce ops cases