Ecommerce ops · Production

Building DoorDash's Product Knowledge Graph with Large Language Models

The problem

DoorDash's SKU enrichment process was manual and led by contract operators, producing long turnaround times, high costs, and so many inaccuracies that a second human had to audit results. Brand ingestion was reactive and purely manual, limiting volume, failing to close coverage gaps, and generating duplicate brands. Building an in-house extraction model was blocked by the NLP cold-start problem requiring large labeled datasets.

First attempt

Manual SKU enrichment by contract operators was so inaccurate it required a second human audit pass, and the cold-start problem of NLP caused data collection to slow model development and delay adding new items to the active catalog.

Workflow diagram · grounded in source
1
Merchant SKU data ingested
trigger
“When a merchant comes onboard at DoorDash, we add their internal SKU data - raw merchant data - to our retail catalog”
2
In-house classifier tags brands
ai_action
“Unstructured product description is passed to our in-house brand classifier”
3
LLM extracts new brands
ai_action
“SKUs that cannot be tagged confidently to one of the existing brands are passed to an LLM for brand extraction”
4
Second LLM deduplicates brands
validation
“The extraction output is passed to a second LLM, which retrieves similar brands and example item names from an internal knowledge graph to decide whether the extracted brand is a duplicate entity”
5
Classifier retrained with new brands
feedback_loop
“The new brand enters our knowledge graph and the in-house classifier is retrained with the new annotations”
6
LLM organic product labeling
ai_action
“We leverage LLMs to determine whether a product is organic based on available product information. This information could come directly from merchants or via optical character recognition extraction from packaging photos. This approach i…”
7
RAG retrieves similar annotated SKUs
ai_action
“For each unannotated SKU, we first leverage OpenAI embeddings and the approximate nearest neighbors technique to retrieve the most similar SKUs from our golden annotation set”
8
GPT-4 generates attribute annotations
ai_action
“We pass these golden annotation examples to GPT-4 as in-context examples to generate labels for the unannotated SKU”
9
LLM fine-tuned for scalable inference
feedback_loop
“the generated annotations are used to fine-tune an LLM for more scalable inference”
Reported outcome

LLM-powered pipelines enabled proactive brand identification at scale with improved efficiency and accuracy, organic label coverage sufficient to launch personalized item carousels that improved top-line engagement metrics, and attribute annotation generation within a week that would otherwise require months to collect.

Reported metrics
SKU enrichment turnaround timelong turnaround times
SKU enrichment costshigh costs
Organic label precision vs humanbetter than human precision
Annotation generation timewithin a week that would otherwise require months
Show all 5 reported metrics
SKU enrichment turnaround timelong turnaround times
SKU enrichment costshigh costs
organic label precision vs humanbetter than human precision
annotation generation timewithin a week that would otherwise require months
top-line engagementimproved our top-line engagement metrics
Reported stack
GPT-4OpenAI embeddingsRAGOCR
Source
https://doordash.engineering/2024/04/23/building-doordashs-product-knowledge-graph-with-large-language-models/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LLM-powered pipelines enabled proactive brand identification at scale with improved efficiency and accuracy, organic label coverage sufficient to launch personalized item carousels that improved top-line engagement me…

What tools did this team use?

GPT-4, OpenAI embeddings, RAG, OCR.

What results were reported?

SKU enrichment turnaround time: long turnaround times; SKU enrichment costs: high costs; Organic label precision vs human: better than human precision; Annotation generation time: within a week that would otherwise require months (source-reported, not independently verified).

What failed first in this deployment?

Manual SKU enrichment by contract operators was so inaccurate it required a second human audit pass, and the cold-start problem of NLP caused data collection to slow model development and delay adding new items to the…

How is this ecommerce ops AI workflow structured?

Merchant SKU data ingested → In-house classifier tags brands → LLM extracts new brands → Second LLM deduplicates brands → Classifier retrained with new brands → LLM organic product labeling → RAG retrieves similar annotated SKUs → GPT-4 generates attribute annotations → LLM fine-tuned for scalable inference.

Related ecommerce ops cases