Compliance monitoring · Production

How Roblox Uses AI to Moderate Content on a Massive Scale

The problem

Roblox's user-generated platform grew in both scale and speed far beyond what human moderators could handle alone, requiring scalable AI infrastructure to moderate billions of pieces of content in real time across dozens of languages.

First attempt

An earlier rules-based text filter and CPU-based serving infrastructure could not keep pace with the volume and speed demands of the platform as it scaled.

Workflow diagram · grounded in source
1
Content submitted by users
Trigger
Users send chat messages, voice communications, and upload assets to the platform every day.
▾ source quote
“Every day, users send an average of 6.1 billion chat messages and 1.1 million hours of voice communication in 28 different languages. Creators upload millions of assets per day”
2
ML text filtering
Ai action
Transformer-based ML models filter text for policy violations—including PII, profanity, and hate speech—within milliseconds.
▾ source quote
“our text filters process an average of 6.1 billion chat messages per day, powered by many models that are purpose-built for different types of policy violations”
3
Voice safety classification
Ai action
The voice safety classifier moderates voice chat within 15 seconds across eight languages.
▾ source quote
“Our voice safety classifier moderates chat within 15 seconds across eight languages”
4
Real-time user feedback
Output
Warning notifications, time-outs, and suspensions are issued to users in real time to educate and change behavior.
▾ source quote
“Our multilayered defense system includes proactive measures like warning notifications, time-outs, and suspensions”
5
Human expert review
Human review
Human experts handle evolving and rare cases, complex investigations, and appeals.
▾ source quote
“We leverage humans for continuously improving AI, evolving and rare cases, complex investigations, and appeals”
6
Active learning update
Feedback loop
Active learning systems continuously update models as language evolves, user patterns change, and real-world events happen.
▾ source quote
“Our active learning systems continuously update models as language evolves, user patterns change, and real-world events happen”
Reported outcome

AI moderation now handles over 750,000 text filter RPS and 370,000 PII filter RPS at peak.
The PII filter reduced false positives by 30% and increased automatic PII detection by 25%. The voice safety classifier achieves 92% higher recall than the initial version at a 1% false positive rate. Real-time feedback interventions reduced filtered chat messages by 5% and abuse-report consequences by 6%.

Reported metrics
Daily chat messages6.1 billion
Daily voice communication1.1 million hours
Policy violation detection rate0.01%
Median time to action for illegal contentten minutes
Show all 17 reported metrics
daily chat messages6.1 billion
daily voice communication1.1 million hours
policy violation detection rate0.01%
median time to action for illegal contentten minutes
PII filter peak RPS370,000 RPS
PII filter false positive reduction30%
PII detection increase25%
text filter peak RPSmore than 750,000 RPS
RPS improvement from GPU serving stackquadrupled
voice classifier recall improvement vs initial version92% higher
voice classifier false positive rate1%
voice classifier peak RPSup to 8,300 RPS
voice moderation latencywithin 15 seconds
filtered chat messages reduction from real-time feedback5%
abuse report consequences reduction from real-time feedback6%
user adoption of visual annotations in abuse reportsapproximately 15%
suspension impact duration on reoffenseup to three weeks
Reported stack
transformer-based text filterPII filtervoice safety classifierlarge language models (LLMs)
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://corp.roblox.com/newsroom/2025/07/roblox-ai-moderation-massive-scale
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

AI moderation now handles over 750,000 text filter RPS and 370,000 PII filter RPS at peak.

What tools did this team use?

transformer-based text filter, PII filter, voice safety classifier, large language models (LLMs).

What results were reported?

Daily chat messages: 6.1 billion; Daily voice communication: 1.1 million hours; Policy violation detection rate: 0.01%; Median time to action for illegal content: ten minutes (source-reported, not independently verified).

What failed first in this deployment?

An earlier rules-based text filter and CPU-based serving infrastructure could not keep pace with the volume and speed demands of the platform as it scaled.

How is this compliance monitoring AI workflow structured?

Content submitted by users → ML text filtering → Voice safety classification → Real-time user feedback → Human expert review → Active learning update.

Related compliance monitoring cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Compliance monitoring cases →