Customer support · Production

DoorDash builds simulation and evaluation flywheel to develop LLM support chatbots at scale

The problem

LLMs' non-determinism made safe testing of support chatbot changes impossible: deploying to production risked degrading customer and Dasher experience, while manual testing was too slow and likely to miss problems.

First attempt

The early LLM implementation suffered from hallucinations because the context window was overwhelmed with raw events and logs, causing the model to misinterpret fields or suggest non-existent policies; iterative attempts at summarization either lost important details or remained too noisy.

Workflow diagram · grounded in source
1
Identify customer problem
trigger
“we continue to prioritize manual review of cases, either from an early simulation if we're building a new automation, or from actual users if a process has already been deployed”
2
Generate test scenarios from transcripts
ai_action
“LLMs analyze historical conversations from our database, extracting comprehensive user behavior insights, including: - Customer characteristics — "frustrated, demanding, direct" vs. "confused, polite, patient" - Customer story — detailed…”
3
Simulate multi-turn customer conversations
ai_action
“The simulator is responsible for playing the customer's role in conversations, but it doesn't provide simple scripted responses. Instead, it uses LLMs with detailed decision-making prompts to generate dynamic, realistic customer behavior…”
4
LLM-as-judge evaluation
ai_action
“The evaluation framework uses LLM-as-a-judge as a proxy for human reviewers. Because manually reading hundreds of simulated conversations defeats the purpose of automation, we developed calibrated evaluations to match expert human judgment”
5
Calibrate against human judgment
validation
“Calculate precision, recall, and F1 scores against human labels.”
6
Iterate on the system
feedback_loop
“We then baseline the current system; for example, it may pass 50% of test cases. We can then modify the prompt, run the simulator, and recheck the evaluation. If the pass rate in our example climbs to 60%, we know we're moving in the rig…”
7
Validate guardrails and deploy
output
“If no degradation is detected, and all guardrail pass rates remain stable, then we can deploy the changes into our production system via our standard A/B test”
Reported outcome

The flywheel reduced hallucinations by 90% in simulation with the improvement carrying over into production, cut each iteration cycle from days to hours, and enabled more than 200 simulated conversations to run in under five minutes.

Reported metrics
Hallucination reduction in simulations90%
Iteration cycle timereduced each iteration cycle from days to hours
Simulated conversations per runmore than 200 in under five minutes
Evaluation suite sizemore than 50 evaluations
Reported stack
LLMsS3gRPC
Source
https://careersatdoordash.com/blog/doordash-simulation-evaluation-flywheel-to-develop-llm-chatbots-at-scale/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The flywheel reduced hallucinations by 90% in simulation with the improvement carrying over into production, cut each iteration cycle from days to hours, and enabled more than 200 simulated conversations to run in und…

What tools did this team use?

LLMs, S3, gRPC.

What results were reported?

Hallucination reduction in simulations: 90%; Iteration cycle time: reduced each iteration cycle from days to hours; Simulated conversations per run: more than 200 in under five minutes; Evaluation suite size: more than 50 evaluations (source-reported, not independently verified).

What failed first in this deployment?

The early LLM implementation suffered from hallucinations because the context window was overwhelmed with raw events and logs, causing the model to misinterpret fields or suggest non-existent policies; iterative attem…

How is this customer support AI workflow structured?

Identify customer problem → Generate test scenarios from transcripts → Simulate multi-turn customer conversations → LLM-as-judge evaluation → Calibrate against human judgment → Iterate on the system → Validate guardrails and deploy.

Related customer support cases