Compliance monitoring · Production

Yelp deploys fine-tuned LLMs to proactively detect inappropriate reviews in real time

The problem

Yelp needed to detect hate speech, lewdness, threats, and other inappropriate content in user reviews at scale and in real time, requiring high precision to avoid delaying legitimate reviews while preventing harmful content from being published.

First attempt

Prior automated approaches to inappropriate content detection had unsatisfactory tradeoffs between precision and recall, prompting iteration toward LLMs.

Workflow diagram · grounded in source
1
Moderator dataset curation
human_review
“we collaborated with Yelp's User Operations team to curate a high-quality dataset comprising the most egregious instances of inappropriate reviews, as well as reviews that adhered to our content guidelines. A pivotal strategy here was th…”
2
Similarity-based dataset augmentation
ai_action
“we also implemented similarity techniques using sentence embeddings from LLMs, and identified additional reviews that were similar to the high-quality samples we obtained from moderator annotation”
3
Embedding cluster analysis
ai_action
“We downloaded LLMs from the HuggingFace model hub and computed sentence embeddings on the preprocessed review samples. Using these embeddings, we determined the separation between appropriate and inappropriate samples by evaluating the s…”
4
LLM fine-tuning for classification
ai_action
“we minimally fine-tuned the same model on the dataset for the given classification task and saw successful results on the class-balanced dataset”
5
Mock traffic threshold calibration
validation
“we generated many sets of mock traffic data with different degrees of spam prevalence. The result of this analysis allowed us to determine the model threshold at which we can identify inappropriate reviews with an accepted range of confi…”
6
Real-time review classification
ai_action
“classify reviews as appropriate or inappropriate, in real-time”
7
Human review of flagged content
human_review
“The flagged reviews are manually reviewed by our User Operations team”
8
Moderator-driven model retraining
feedback_loop
“Based on the decisions made by moderators and subsequent retraining of the model, we anticipate further improvements in the model's recall in the future”
Reported outcome

Since deploying the LLM pipeline, Yelp's moderators proactively prevented 23,600+ reviews from ever publishing to the platform in 2023, with ongoing moderator feedback expected to further improve the model's recall.

Reported metrics
Reviews proactively prevented from publishing23,600+
User-reported reviews removed in 202226,500+
Reported stack
Large Language ModelsHuggingFace model hubRedshiftS3MLFlowMLeapt-SNE
Source
https://engineeringblog.yelp.com/2024/03/ai-pipeline-inappropriate-language-detection.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Since deploying the LLM pipeline, Yelp's moderators proactively prevented 23,600+ reviews from ever publishing to the platform in 2023, with ongoing moderator feedback expected to further improve the model's recall.

What tools did this team use?

Large Language Models, HuggingFace model hub, Redshift, S3, MLFlow, MLeap, t-SNE.

What results were reported?

Reviews proactively prevented from publishing: 23,600+; User-reported reviews removed in 2022: 26,500+ (source-reported, not independently verified).

What failed first in this deployment?

Prior automated approaches to inappropriate content detection had unsatisfactory tradeoffs between precision and recall, prompting iteration toward LLMs.

How is this compliance monitoring AI workflow structured?

Moderator dataset curation → Similarity-based dataset augmentation → Embedding cluster analysis → LLM fine-tuning for classification → Mock traffic threshold calibration → Real-time review classification → Human review of flagged content → Moderator-driven model retraining.

Related compliance monitoring cases