compliance_monitoring · ecommerce · workflow
How Whatnot Utilizes Generative AI to Enhance Trust and Safety
As Whatnot's marketplace grew, scam attempts increasingly targeted new users through multi-message confidence-building exchanges that existing rule-based and single-message ML approaches could not reliably detect due to a lack of contextual understanding.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · User signal qualification
User signals such as messaging patterns and account age are used as qualifiers to determine which messages should be analyzed through LLMs.
Tools used
LLMsOCRKafka
Outcome
LLM-based automated detection proactively identifies over 95% of scam attempts on the platform within a few minutes, with 96% precision and high recall in LLM output.
What failed first
The centralized rule engine was limited to scalar values and could not reason about conversational context, and traditional ML content moderation assessed messages in isolation, making both approaches ineffective for detecting multi-turn scam or harassment patterns.
Results
Time savedwithin a few minutes
Volumeover 95%
Grounding & classification
Source type: technical build writeup
28 fields verified against source quotes.
fraud detectionocrchat transcriptbuilder submittedfailure mode describedhuman review describedmetric backednamed customerproduction runtime claimedsource backedtools describedworkflow describedecommerceaccuracy improvementautomation rateresponse time reductiontechnical build writeupcompliance monitoringquality assuranceescalation workflowextract classify routehuman review queue