Compliance monitoring · Production

DoorDash SafeChat cuts safety incidents 50% with multi-modal AI content moderation

The problem

DoorDash needed to screen a high volume of user-generated content daily — chat messages, images, and voice calls — for inappropriate material to protect Dashers and consumers at platform scale.

First attempt

An initial shadow test with an external vision API for image moderation produced a significant false positive rate, making it unsuitable for production.

Workflow diagram · grounded in source
1
User-generated content enters pipeline
trigger
“Content moderation is performed for every piece of content exchanged between parties in the system. Its architecture differs depending on the content type — text, image, or voice.”
2
Layer 1 fast model screening
ai_action
“For each incoming message, it produces two values: safeScore and unsafeScore. These always sum to 1. If the safeScore is high, the message is automatically cleared. If the unsafeScore is high, the message is considered suspicious, requir…”
3
Layer 2 precise LLM evaluation
ai_action
“Only a small percentage of messages flagged by the internal model move to the precise LLM. Because this layer is resource-intensive, we reserve it for ambiguous or high-risk cases.”
4
Computer vision image moderation
ai_action
“We then explored other computer vision models with greater granularity. Unlike the initial API, which only returned broad categories, the new model that we selected produced a richer set of subcategories, each with a confidence score. Th…”
5
Real-time voice call moderation
ai_action
“Our system processes calls in real time, analyzing voice tone, spoken words, and conversational context across multiple languages. Each call is streamed into the moderation pipeline, where it is labeled under categories with confidence s…”
6
Severity-based action routing
routing
“When unsafe content is detected, the system decides how to act. The response depends on the severity of the offense and the risk involved.”
7
Human safety agent escalation
human_review
“Severe or repeated offenses are reviewed by human safety agents. Based on severity, recurrence, and context, actions may include temporary suspension or permanent removal from the platform.”
8
Proportional enforcement output
output
“By layering responses, we can act proportionally. The system can censor a single message or block repeat offenders. This ensures interventions are fast and effective.”
Reported outcome

Since launch, SafeChat achieved a 50% decrease in low/medium-severity safety incidents and 99.8% of messages are now handled by the internal model alone, resulting in higher Dasher and consumer trust, reduced customer support volume, and stronger platform reliability.

Reported metrics
Low/medium-severity safety incidents50%
messages handled by internal model alone (Phase 2)99.8%
messages auto-cleared by Moderation API (Phase 1 Layer 1)about 90%
messages identified as safe at Phase 1 Layer 2 cutoff99.8%
Show all 7 reported metrics
low/medium-severity safety incidents50%
messages handled by internal model alone (Phase 2)99.8%
messages auto-cleared by Moderation API (Phase 1 Layer 1)about 90%
messages identified as safe at Phase 1 Layer 2 cutoff99.8%
Layer 1 text moderation response timeunder 300ms
image moderation latency per imageabout 500ms
training data points accumulated for internal modelaround 10 million
Reported stack
moderation APIfast LLMprecise LLMinternal modelcomputer vision
Source
https://careersatdoordash.com/blog/doordash-safechat-ai-safety-feature/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Since launch, SafeChat achieved a 50% decrease in low/medium-severity safety incidents and 99.8% of messages are now handled by the internal model alone, resulting in higher Dasher and consumer trust, reduced customer…

What tools did this team use?

moderation API, fast LLM, precise LLM, internal model, computer vision.

What results were reported?

Low/medium-severity safety incidents: 50%; messages handled by internal model alone (Phase 2): 99.8%; messages auto-cleared by Moderation API (Phase 1 Layer 1): about 90%; messages identified as safe at Phase 1 Layer 2 cutoff: 99.8% (source-reported, not independently verified).

What failed first in this deployment?

An initial shadow test with an external vision API for image moderation produced a significant false positive rate, making it unsuitable for production.

How is this compliance monitoring AI workflow structured?

User-generated content enters pipeline → Layer 1 fast model screening → Layer 2 precise LLM evaluation → Computer vision image moderation → Real-time voice call moderation → Severity-based action routing → Human safety agent escalation → Proportional enforcement output.

Related compliance monitoring cases