Customer support · Production

DoorDash improves Dasher support with LLM-based RAG chatbot, reducing hallucinations by 90%

The problem

DoorDash's existing automated support relied on rigid flow-based resolution paths that could address only a small subset of Dasher issues. The knowledge base was also hard to navigate, time-consuming to read, and available only in English despite many Dashers preferring other languages.

First attempt

The initial LLM RAG chatbot produced responses that diverged from the intended knowledge base context, with LLMs drawing on potentially erroneous public sources such as Quora, Reddit, and Twitter rather than grounded DoorDash information.

Workflow diagram · grounded in source
1
Dasher submits issue
trigger
“The process, as outlined in Figure 2, begins when a Dasher presents an issue to the chatbot”
2
Conversation summarization
ai_action
“Because the issue likely will be spread across several messages and follow-up questions, the system first condenses the entire conversation to pinpoint the core problem”
3
KB article retrieval
ai_action
“Using this summary, it then searches historical data for the top N similar cases previously resolved with information from KB articles. Each identified issue corresponds to a specific article that is integrated into the prompt template”
4
Tailored response generation
ai_action
“This enriched template allows the chatbot to generate a tailored response, leveraging the context of the conversation, the distilled issue summary, and any relevant KB articles to ensure that Dashers receive precise and informed support”
5
LLM Guardrail validation
validation
“The LLM Guardrail system is an online monitoring tool that evaluates each output from the LLM to ensure accuracy and compliance. It checks the grounding of RAG information to prevent hallucinations, maintains response coherence with prev…”
6
Human agent fallback
routing
“strategically defaulting to human agents can be an effective way to ensure a quality user experience while maintaining a high level of automation”
7
LLM Judge quality monitoring
feedback_loop
“The overall quality of each aspect is determined by prompting LLM Judge with open-ended questions, as shown in Figure 4. Answers to these questions are processed and summarized into common issues. The high-frequency issues are then built…”
8
Human review calibration
human_review
“we also have a dedicated human team that reviews random subset transcript samples. A continuous calibration between this human review and the automated system ensures effective coverage”
Reported outcome

The LLM Guardrail reduced hallucinations by 90% and compliance issues by 99%, and the chatbot now autonomously assists thousands of Dashers each day while freeing human agents to focus on more complex problems.

Reported metrics
Hallucinations reduced90%
Compliance issues reduced99%
Dashers assisted dailythousands of Dashers
Human agent focus shift to complex problemsfocus their energy on solving more complex problems
Reported stack
RAGGPT-4Claude-3LLM GuardrailLLM JudgePromptfoovector store
Source
https://careersatdoordash.com/blog/large-language-modules-based-dasher-support-automation/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM Guardrail reduced hallucinations by 90% and compliance issues by 99%, and the chatbot now autonomously assists thousands of Dashers each day while freeing human agents to focus on more complex problems.

What tools did this team use?

RAG, GPT-4, Claude-3, LLM Guardrail, LLM Judge, Promptfoo, vector store.

What results were reported?

Hallucinations reduced: 90%; Compliance issues reduced: 99%; Dashers assisted daily: thousands of Dashers; Human agent focus shift to complex problems: focus their energy on solving more complex problems (source-reported, not independently verified).

What failed first in this deployment?

The initial LLM RAG chatbot produced responses that diverged from the intended knowledge base context, with LLMs drawing on potentially erroneous public sources such as Quora, Reddit, and Twitter rather than grounded…

How is this customer support AI workflow structured?

Dasher submits issue → Conversation summarization → KB article retrieval → Tailored response generation → LLM Guardrail validation → Human agent fallback → LLM Judge quality monitoring → Human review calibration.

Related customer support cases