Ecommerce ops · Production

DoorDash uses LLMs and RAG to build a Product Knowledge Graph and supercharge search for New Verticals

The problem

DoorDash's expansion into new verticals — groceries, alcohol, and retail — created the challenge of handling hundreds of thousands of SKUs requiring accurate product attribute extraction and catalog management. Traditional human annotation for training ML models was time-consuming and expensive, and a cold start problem made it difficult to quickly launch model coverage for new product categories.

First attempt

Traditional human annotation workflows were expensive and slow, unable to scale quickly to new product categories. Engagement-based training signals for search were noisy and sparse for niche tail queries, limiting search relevance model quality.

Workflow diagram · grounded in source
1
Create golden annotations
human_review
“We begin by creating a few high-quality, manually labeled annotations—known as "golden" annotations—for new categories or products”
2
RAG generates silver annotations
ai_action
“using Retrieval-Augmented Generation (RAG), we generate many additional "silver" annotations”
3
Fine-tune attribute extraction model
ai_action
“This expanded dataset allows us to fine-tune LLMs, resulting in a Generalized Attribute Extraction model capable of identifying and extracting critical product attributes across diverse categories”
4
LLM catalog inconsistency detection
ai_action
“Our process begins by constructing a natural language prompt based on primary attributes like the item name, photo, and unit information. The LLM evaluates the product details, identifying any discrepancies between the listed attributes …”
5
Classify and route catalog issues
routing
“the system classifies the issue into priority buckets — P0, P1, or P2 — based on the severity and urgency of the fix”
6
LLM tail query labeling
ai_action
“we leverage Large Language Models (LLMs) to improve training data quality at scale. LLMs help us assign labels to these less common queries, enhancing the accuracy and reliability of our search systems”
7
Personalized search ranking
ai_action
“we utilize personalization in search to rank results based on individual preferences such as dietary needs, brand affinities, price sensitivity, and shopping habits”
8
Consensus relevance labeling
ai_action
“By implementing consensus labeling with LLMs, we ensure our search engine remains precise and capable of consistently delivering the most relevant results to our users”
Reported outcome

LLM-assisted annotation significantly reduced costs and enabled model training in days instead of weeks.
The approach enhanced product discovery, enabled quicker adaptation to new verticals, and delivered a smoother, more trustworthy shopping experience while reducing human annotator workload.

Reported metrics
Model training timetrain models in days instead of weeks
Annotation costssignificantly reducing costs
Human annotator workloadreduce the workload for our human annotators
Model development speedspeeds up the model development process
Reported stack
Large Language ModelsRAGNLPRayLoRAQLoRA
Source
https://careersatdoordash.com/blog/unleashing-the-power-of-large-language-models-at-doordash-for-a-seamless-shopping-adventure/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LLM-assisted annotation significantly reduced costs and enabled model training in days instead of weeks.

What tools did this team use?

Large Language Models, RAG, NLP, Ray, LoRA, QLoRA.

What results were reported?

Model training time: train models in days instead of weeks; Annotation costs: significantly reducing costs; Human annotator workload: reduce the workload for our human annotators; Model development speed: speeds up the model development process (source-reported, not independently verified).

What failed first in this deployment?

Traditional human annotation workflows were expensive and slow, unable to scale quickly to new product categories.

How is this ecommerce ops AI workflow structured?

Create golden annotations → RAG generates silver annotations → Fine-tune attribute extraction model → LLM catalog inconsistency detection → Classify and route catalog issues → LLM tail query labeling → Personalized search ranking → Consensus relevance labeling.

Related ecommerce ops cases