Customer support · Production

DoorDash 2025 Summer Interns Build In-House LLM for Never-Delivered Order Feature Extraction and RAG-Based Chatbot Service

The problem

DoorDash's never-delivered order review process was fully manual, slow, and expensive, limiting resolutions to just a few cases per day. Separately, there was no centralized platform to manage knowledge bases or a unified API to deploy chatbots across internal teams.

First attempt

Teams previously stored article embeddings in a vector database while managing metadata separately in spreadsheets, creating an error-prone two-step lookup workflow that did not scale.

Workflow diagram · grounded in source
1
ND order reported
trigger
“A never-delivered designation, or ND, occurs when a customer reports an order was not received, despite the Dasher marking the order as delivered.”
2
LLM extracts conversation features
ai_action
“we developed a custom in-house language model that automatically extracts key features from these conversations, accelerating the review process and improving our downstream attribution model”
3
Model predicts resolution answers
ai_action
“the model steps through the resolution process, predicting viable answers at each juncture”
4
Features passed to ND model
output
“Answers are passed as a feature to the downstream ND model”
5
Backfilling improves ND model
feedback_loop
“This enables rapid backfilling and helps the ND model improve over time”
6
Knowledge articles ingested and embedded
integration
“Teams upload their content through the knowledge base management API. The platform automatically handles embedding generation and metadata storage”
7
RAG API generates chatbot response
ai_action
“the API retrieves relevant knowledge base articles, combines them with the user's conversation history and a customizable prompt, and generates a context-aware response”
Reported outcome

The in-house fine-tuned DistilBertForSequenceClassification model achieved an F1 score of 0.8289 and accuracy of 0.9870, automating ND feature extraction and reducing investigation costs.
A centralized RAG-based chatbot service now powers the Dasher-facing assistant.

Reported metrics
ND LLM F1 score (balanced DistilBert)0.8289
ND LLM accuracy (balanced DistilBert)0.9870
ND LLM model latency0.0936 seconds
Investigation costs and response timesreduce investigation costs and speed up response times
Show all 13 reported metrics
ND LLM F1 score (balanced DistilBert)0.8289
ND LLM accuracy (balanced DistilBert)0.9870
ND LLM model latency0.0936 seconds
Investigation costs and response timesreduce investigation costs and speed up response times
Manual ND resolutions before automationjust a few cases each day
New banner/tag deployment time — before SDUI2-3 days
New banner/tag deployment time — after SDUI<1 day
Banner/tag modification time — before SDUI1 day
Banner/tag modification time — after SDUI<1 hour
Unusable or missing store tags weekly (legacy)500 to 1,500 unusable or missing tags weekly
Annotation pipeline throughputone to two requests per second
Annotation pipeline P99 latencythree seconds
New CRDB records from annotation pipelineover 1,000 new records
Reported stack
Meta Llama 3DistilBertForSequenceClassificationKafkaCadenceCockroachDBAppenRAGvector databaseMosaicKotlin
Source
https://careersatdoordash.com/blog/part-1-doordash-2025-summer-intern-projects/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The in-house fine-tuned DistilBertForSequenceClassification model achieved an F1 score of 0.8289 and accuracy of 0.9870, automating ND feature extraction and reducing investigation costs.

What tools did this team use?

Meta Llama 3, DistilBertForSequenceClassification, Kafka, Cadence, CockroachDB, Appen, RAG, vector database, Mosaic, Kotlin.

What results were reported?

ND LLM F1 score (balanced DistilBert): 0.8289; ND LLM accuracy (balanced DistilBert): 0.9870; ND LLM model latency: 0.0936 seconds; Investigation costs and response times: reduce investigation costs and speed up response times (source-reported, not independently verified).

What failed first in this deployment?

Teams previously stored article embeddings in a vector database while managing metadata separately in spreadsheets, creating an error-prone two-step lookup workflow that did not scale.

How is this customer support AI workflow structured?

ND order reported → LLM extracts conversation features → Model predicts resolution answers → Features passed to ND model → Backfilling improves ND model → Knowledge articles ingested and embedded → RAG API generates chatbot response.

Related customer support cases