Customer support · Production

DoorDash pairs clustering algorithms and LLMs to auto-identify knowledge base gaps and draft support articles

The problem

As DoorDash's marketplace grew, manually maintaining the support chatbot knowledge base became unscalable — new policies, product changes, and a long tail of edge cases demanded constant fresh answers that were too resource-intensive and time-consuming to handle by hand.

Workflow diagram · grounded in source
1
Escalated chat ingestion
trigger
“we begin by feeding thousands of anonymized chat transcripts into a semantic clustering pipeline, selecting only those conversations that were escalated to a live agent so that we can zero in on the cases where our chatbot fell short”
2
Semantic clustering of chats
ai_action
“every chat summary we use is run through an open-source embedding model, chosen for its strong performance in semantic-similarity tasks. Those vectors flow into a lightweight clustering routine... we measure its cosine similarity to all …”
3
LLM classification and draft generation
ai_action
“These high‑ROI topics then pass through an LLM that simultaneously tackles two jobs: Smart classifier: This classifies each cluster as either an actionable problem... or an informational query... First‑draft generation: For each informat…”
4
Human review and refinement
human_review
“Each auto‑draft flows into a lightweight review queue where content specialists and our operations partners sanity‑check policy references, tone, and edge cases... Reviewers flag these nuances and either spin off tailored variants or ann…”
5
RAG-powered article serving
output
“Once approved, articles are surfaced by the chatbot via a RAG layer... The chatbot now retrieves the right article, blends it with conversation history and context, and answers with accurate and timely information”
6
LLM judge and A/B testing
feedback_loop
“Offline experiments using an LLM judge are conducted to benchmark improvements over existing KB articles... Online A/B testing with selected audiences is conducted to assess impact”
Reported outcome

The system lowers escalation rates, improves customer satisfaction, and reduces specialist manual workload.
In A/B tests, high-traffic escalation clusters saw escalation rates drop from 78% in the control group to 43% in the treatment group, and roughly 75% of KB retrieval events in the treatment group contained only UGC KB content.

Reported metrics
Escalation rate — control group78%
Escalation rate — treatment group43%
KB retrieval events containing only UGC KB contentroughly 75%
KB article drafting speedminutes instead of weeks
Show all 7 reported metrics
escalation rate — control group78%
escalation rate — treatment group43%
KB retrieval events containing only UGC KB contentroughly 75%
KB article drafting speedminutes instead of weeks
KB article edit timeminutes instead of days
customer satisfactionimproving customer satisfaction
specialist manual workloadfreeing our specialists from manual transcript review
Reported stack
large language models (LLMs)clustering algorithmsopen-source embedding modelRAGLLM judge
Source
https://careersatdoordash.com/blog/doordash-llm-chatbot-knowledge-with-ugc/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The system lowers escalation rates, improves customer satisfaction, and reduces specialist manual workload.

What tools did this team use?

large language models (LLMs), clustering algorithms, open-source embedding model, RAG, LLM judge.

What results were reported?

Escalation rate — control group: 78%; Escalation rate — treatment group: 43%; KB retrieval events containing only UGC KB content: roughly 75%; KB article drafting speed: minutes instead of weeks (source-reported, not independently verified).

How is this customer support AI workflow structured?

Escalated chat ingestion → Semantic clustering of chats → LLM classification and draft generation → Human review and refinement → RAG-powered article serving → LLM judge and A/B testing.

Related customer support cases