Compliance monitoring · Production

How Whatnot Utilizes Generative AI to Enhance Trust and Safety

The problem

As Whatnot's marketplace grew, scam attempts increasingly targeted new users through multi-message confidence-building exchanges that existing rule-based and single-message ML approaches could not reliably detect due to a lack of contextual understanding.

First attempt

The centralized rule engine was limited to scalar values and could not reason about conversational context, and traditional ML content moderation assessed messages in isolation, making both approaches ineffective for detecting multi-turn scam or harassment patterns.

Workflow diagram · grounded in source
1
User signal qualification
trigger
“We use different user signals (messaging patterns, account age) as qualifiers to determine which messages should be analyzed through LLMs.”
2
Multi-source data curation
integration
“we curate data from various sources like (events, user data, order history, ML models). This phase includes data identification, filtering, annotation, and formatting.”
3
LLM scam likelihood scoring
ai_action
“Provide likelihoods (0-1) of scam, assessment notes in json format which can be consumed by a service with keys with no text output: scam_likelihood and explanation (reasoning for the likelihood)”
4
Rule engine threshold check
validation
“scam_likelihood > 0.6 and account_age < X days and message_frequency > Y and lifetime_orders < Z”
5
Ops team notification and review
human_review
“we take temp action to disable certain features on the app and notify our ops team and pass along the LLM output (with likelihood and explanation) to investigate/action the user”
6
Enforcement via Kafka
output
“the user is notified of the violation and the system is updated to reflect product access changes (if any) through Kafka”
Reported outcome

LLM-based automated detection proactively identifies over 95% of scam attempts on the platform within a few minutes, with 96% precision and high recall in LLM output.

Reported metrics
Scam attempts proactively detectedover 95%
LLM output precision96%
LLM output recallhigh recall
Detection timewithin a few minutes
Reported stack
LLMsOCRKafka
Source
https://medium.com/whatnot-engineering/how-whatnot-utilizes-generative-ai-to-enhance-trust-and-safety-c7968eb6315e
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LLM-based automated detection proactively identifies over 95% of scam attempts on the platform within a few minutes, with 96% precision and high recall in LLM output.

What tools did this team use?

LLMs, OCR, Kafka.

What results were reported?

Scam attempts proactively detected: over 95%; LLM output precision: 96%; LLM output recall: high recall; Detection time: within a few minutes (source-reported, not independently verified).

What failed first in this deployment?

The centralized rule engine was limited to scalar values and could not reason about conversational context, and traditional ML content moderation assessed messages in isolation, making both approaches ineffective for…

How is this compliance monitoring AI workflow structured?

User signal qualification → Multi-source data curation → LLM scam likelihood scoring → Rule engine threshold check → Ops team notification and review → Enforcement via Kafka.

Related compliance monitoring cases