Customer support · Production

DoorDash deploys LLM-based RAG chatbot with guardrail and quality monitoring to autonomously support Dashers

The problem

DoorDash's existing automated Dasher support relied on flow-based resolution paths that could address only a small subset of issues. The knowledge base was hard to navigate, time-consuming to read, and available only in English despite many Dashers preferring other languages.

First attempt

The initial LLM RAG chatbot generated responses that diverged from the intended context, appearing natural but potentially inaccurate. Additional challenges included language consistency failures and a requirement for highly accurate conversation summarization before retrieval could function correctly.

Workflow diagram · grounded in source
1
Dasher presents issue
trigger
“The process, as outlined in Figure 2, begins when a Dasher presents an issue to the chatbot.”
2
Conversation condensed to core problem
ai_action
“Because the issue likely will be spread across several messages and follow-up questions, the system first condenses the entire conversation to pinpoint the core problem.”
3
Historical similar cases retrieved
ai_action
“it then searches historical data for the top N similar cases previously resolved with information from KB articles”
4
Tailored response generated
ai_action
“This enriched template allows the chatbot to generate a tailored response, leveraging the context of the conversation, the distilled issue summary, and any relevant KB articles to ensure that Dashers receive precise and informed support.”
5
LLM Guardrail validates response
validation
“The LLM Guardrail system is an online monitoring tool that evaluates each output from the LLM to ensure accuracy and compliance. It checks the grounding of RAG information to prevent hallucinations, maintains response coherence with prev…”
6
Human agent fallback routing
routing
“strategically defaulting to human agents can be an effective way to ensure a quality user experience while maintaining a high level of automation”
7
LLM Judge monitors quality
feedback_loop
“The overall quality of each aspect is determined by prompting LLM Judge with open-ended questions. Answers to these questions are processed and summarized into common issues. The high-frequency issues are then built into prompts or rules…”
8
Human team reviews transcripts
human_review
“we also have a dedicated human team that reviews random subset transcript samples. A continuous calibration between this human review and the automated system ensures effective coverage.”
Reported outcome

The LLM Guardrail reduced hallucinations by 90% and cut potentially severe compliance issues by 99%.
The system now autonomously assists thousands of Dashers each day, allowing human agents to focus on more complex problems.

Reported metrics
Hallucinations reduced90%
Potentially severe compliance issues reduced99%
Dashers autonomously assisted per dayautonomously assists thousands of Dashers
Reported stack
LLMsRAGLLM GuardrailLLM Judgevector storePromptfoo
Source
https://careers.doordash.com/blog/large-language-modules-based-dasher-support-automation/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM Guardrail reduced hallucinations by 90% and cut potentially severe compliance issues by 99%.

What tools did this team use?

LLMs, RAG, LLM Guardrail, LLM Judge, vector store, Promptfoo.

What results were reported?

Hallucinations reduced: 90%; Potentially severe compliance issues reduced: 99%; Dashers autonomously assisted per day: autonomously assists thousands of Dashers (source-reported, not independently verified).

What failed first in this deployment?

The initial LLM RAG chatbot generated responses that diverged from the intended context, appearing natural but potentially inaccurate.

How is this customer support AI workflow structured?

Dasher presents issue → Conversation condensed to core problem → Historical similar cases retrieved → Tailored response generated → LLM Guardrail validates response → Human agent fallback routing → LLM Judge monitors quality → Human team reviews transcripts.

Related customer support cases