Compliance monitoring · Production

How Pinterest Built a Real-Time Radar for Violative Content Using AI

The problem

Pinterest's Trust & Safety teams could only measure policy-violating content prevalence through expensive human review studies run roughly every six months. User reports alone were incomplete, missing under-reported harms and lacking statistical power for rare categories.

First attempt

Human-only prevalence studies required at least two independent reviewers per item plus adjudication, were subject to instability, ran infrequently, and produced post-intervention comparisons that were slow and hard to trust.

Workflow diagram · grounded in source
1
Daily impressions stream sampled
trigger
“We sample images from the daily user impressions stream”
2
Risk-score weighted reservoir sampling
ai_action
“using risk scores from our production enforcement models to improve sampling efficiency, not as labels or inclusion criteria (the same models that remove policy-violating content)...missing scores are imputed with the day's median so fre…”
3
Multimodal LLM bulk labeling
ai_action
“We bulk-label the sample with a multimodal LLM (vision + text) using prompts reviewed by policy subject matter experts (SMEs). The system logs decisions, brief rationales, and full lineage (policy version, prompt, and model IDs) for audi…”
4
Pre-launch LLM quality gate
validation
“Before launching to production the LLM must meet a minimum decision quality requirement relative to human review”
5
Human validation of label subsample
human_review
“A random subsample of labels routes to an internal human validation queue for continual checks of the AI's decision quality”
6
Prevalence estimation and dashboard
output
“We compute overall prevalence and pivots, persist estimates, weights, and labels to production stores, and write diagnostics/lineage for audits. The dashboard surfaces the point estimate, 95% CI, CI width, and effective sample size.”
7
SME gold-set drift monitoring
feedback_loop
“LLM + prompt quality is periodically checked against Subject Matter Expert-labeled gold sets (ground truth) to detect model drift and ensure the labeler/classifier remains accurate and aligned with current violations policy. This continu…”
Reported outcome

The AI-assisted workflow enables daily prevalence measurement at 15x faster labeling turnaround and orders of magnitude lower operational cost than a human-only workflow, while preserving comparable decision quality and enabling continuous monitoring with real-time alerting.

Reported metrics
Labeling speed improvement15x faster
Operational cost vs. human-only workfloworders of magnitude lower operational cost
Measurement continuitycontinuous measurement without historical blind spots
Root cause analysis speedfaster root cause analysis when issues emerge
Show all 6 reported metrics
labeling speed improvement15x faster
operational cost vs. human-only workfloworders of magnitude lower operational cost
measurement continuitycontinuous measurement without historical blind spots
root cause analysis speedfaster root cause analysis when issues emerge
historical prevalence study frequencyroughly every six months
reviewers required per item (legacy workflow)at least two independent reviewers per item
Reported stack
multimodal LLM
Source
https://medium.com/pinterest-engineering/how-pinterest-built-a-real-time-radar-for-violative-content-using-ai-d5a108e02ac2
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The AI-assisted workflow enables daily prevalence measurement at 15x faster labeling turnaround and orders of magnitude lower operational cost than a human-only workflow, while preserving comparable decision quality a…

What tools did this team use?

multimodal LLM.

What results were reported?

Labeling speed improvement: 15x faster; Operational cost vs. human-only workflow: orders of magnitude lower operational cost; Measurement continuity: continuous measurement without historical blind spots; Root cause analysis speed: faster root cause analysis when issues emerge (source-reported, not independently verified).

What failed first in this deployment?

Human-only prevalence studies required at least two independent reviewers per item plus adjudication, were subject to instability, ran infrequently, and produced post-intervention comparisons that were slow and hard t…

How is this compliance monitoring AI workflow structured?

Daily impressions stream sampled → Risk-score weighted reservoir sampling → Multimodal LLM bulk labeling → Pre-launch LLM quality gate → Human validation of label subsample → Prevalence estimation and dashboard → SME gold-set drift monitoring.

Related compliance monitoring cases