Customer support · Production

DoorDash 2025 Summer Interns Build In-House LLM for Never-Delivered Order Feature Extraction and RAG-Based Chatbot Service

The problem

DoorDash's never-delivered order review process was fully manual, slow, and expensive, limiting resolutions to just a few cases per day. Separately, there was no centralized platform to manage knowledge bases or a unified API to deploy chatbots across internal teams.

First attempt

Teams previously stored article embeddings in a vector database while managing metadata separately in spreadsheets, creating an error-prone two-step lookup workflow that did not scale.

Workflow diagram · grounded in source
1
ND order reported
Trigger
A never-delivered designation occurs when a customer reports an order was not received despite the Dasher marking it as delivered.
▾ source quote
“A never-delivered designation, or ND, occurs when a customer reports an order was not received, despite the Dasher marking the order as delivered.”
2
LLM extracts conversation features
Ai action
A custom in-house language model automatically extracts key features from Dasher-consumer conversations.
▾ source quote
“we developed a custom in-house language model that automatically extracts key features from these conversations, accelerating the review process and improving our downstream attribution model”
3
Model predicts resolution answers
Ai action
The model steps through the resolution process, predicting viable answers at each juncture.
▾ source quote
“the model steps through the resolution process, predicting viable answers at each juncture”
4
Features passed to ND model
Output
Predicted answers are passed as features to the downstream ND attribution model.
▾ source quote
“Answers are passed as a feature to the downstream ND model”
5
Backfilling improves ND model
Feedback loop
Automation enables rapid backfilling and helps the ND model improve over time.
▾ source quote
“This enables rapid backfilling and helps the ND model improve over time”
6
Knowledge articles ingested and embedded
Integration
Teams upload knowledge articles through the KB management API, which automatically handles embedding generation and metadata storage.
▾ source quote
“Teams upload their content through the knowledge base management API. The platform automatically handles embedding generation and metadata storage”
7
RAG API generates chatbot response
Ai action
The GetRagResult API retrieves relevant knowledge base articles and generates a context-aware response.
▾ source quote
“the API retrieves relevant knowledge base articles, combines them with the user's conversation history and a customizable prompt, and generates a context-aware response”
Reported outcome

The in-house fine-tuned DistilBertForSequenceClassification model achieved an F1 score of 0.8289 and accuracy of 0.9870, automating ND feature extraction and reducing investigation costs.
A centralized RAG-based chatbot service now powers the Dasher-facing assistant.

Reported metrics
ND LLM F1 score (balanced DistilBert)0.8289
ND LLM accuracy (balanced DistilBert)0.9870
ND LLM model latency0.0936 seconds
Investigation costs and response timesreduce investigation costs and speed up response times
Show all 13 reported metrics
ND LLM F1 score (balanced DistilBert)0.8289
ND LLM accuracy (balanced DistilBert)0.9870
ND LLM model latency0.0936 seconds
Investigation costs and response timesreduce investigation costs and speed up response times
Manual ND resolutions before automationjust a few cases each day
New banner/tag deployment time — before SDUI2-3 days
New banner/tag deployment time — after SDUI<1 day
Banner/tag modification time — before SDUI1 day
Banner/tag modification time — after SDUI<1 hour
Unusable or missing store tags weekly (legacy)500 to 1,500 unusable or missing tags weekly
Annotation pipeline throughputone to two requests per second
Annotation pipeline P99 latencythree seconds
New CRDB records from annotation pipelineover 1,000 new records
Reported stack
Meta Llama 3DistilBertForSequenceClassificationKafkaCadenceCockroachDBAppenRAGvector databaseMosaicKotlin
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://careersatdoordash.com/blog/part-1-doordash-2025-summer-intern-projects/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The in-house fine-tuned DistilBertForSequenceClassification model achieved an F1 score of 0.8289 and accuracy of 0.9870, automating ND feature extraction and reducing investigation costs.

What tools did this team use?

Meta Llama 3, DistilBertForSequenceClassification, Kafka, Cadence, CockroachDB, Appen, RAG, vector database, Mosaic, Kotlin.

What results were reported?

ND LLM F1 score (balanced DistilBert): 0.8289; ND LLM accuracy (balanced DistilBert): 0.9870; ND LLM model latency: 0.0936 seconds; Investigation costs and response times: reduce investigation costs and speed up response times (source-reported, not independently verified).

What failed first in this deployment?

Teams previously stored article embeddings in a vector database while managing metadata separately in spreadsheets, creating an error-prone two-step lookup workflow that did not scale.

How is this customer support AI workflow structured?

ND order reported → LLM extracts conversation features → Model predicts resolution answers → Features passed to ND model → Backfilling improves ND model → Knowledge articles ingested and embedded → RAG API generates chatbot response.

Related customer support cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Customer support cases →