Compliance monitoring · Production

Feedzai ScamAlert: Multi-Label Red Flag Detection with LLM Benchmarking for Scam Prevention

The problem

Traditional binary scam detection systems output only a scam likelihood score without explaining why something is risky or providing contextual guidance, leaving users without interpretable insight and prone to distrust when context is missing.

First attempt

The binary classification approach collapses nuanced scam signals into a single score, fails to account for contextual legitimacy signals the user may already hold, and provides no interpretable explanation, undermining user trust.

Workflow diagram · grounded in source
1
User submits screenshot
trigger
“Users submit a screenshot of the suspected scam, and ScamAlert identifies observable red flags”
2
Multimodal model analyzes image
ai_action
“employing a multimodal model to analyze the image and extract key information”
3
Response validation
validation
“The model response is then thoroughly validated by ScamAlert to ensure that the delivered insights are high-confidence and well-founded”
4
Structured red flag output
output
“ScamAlert outputs a structured response with three parts: (1) a list of the detected red flags, using our predefined nomenclature, (2) a short explanation for why each red flag was identified in the given screenshot, and (3) a list of re…”
Reported outcome

ScamAlert provides interpretable red flag detection rather than a binary score, giving users transparency about why something appears suspicious and allowing domain experts to update flag definitions as fraud tactics evolve.

Reported metrics
GPT-5 performance vs. competitorssignificant performance over competitors
Gemini 3 Pro cost vs. GPT-5 at minimum reasoningsignificantly cheaper
Claude Sonnet performance vs. similarly priced modelsdo not match the performance of similarly priced OpenAI or Gemini models
Latency impact of higher reasoning levelssubstantial increases in latency
Show all 5 reported metrics
GPT-5 performance vs. competitorssignificant performance over competitors
Gemini 3 Pro cost vs. GPT-5 at minimum reasoningsignificantly cheaper
Claude Sonnet performance vs. similarly priced modelsdo not match the performance of similarly priced OpenAI or Gemini models
Latency impact of higher reasoning levelssubstantial increases in latency
Gemini 2.5 Pro speed vs. Gemini 3 Pro and GPT-5significantly faster
Reported stack
ScamAlertLLMGPT-5Claude 4Claude 3.7Gemini 3 ProGemini 2.5 ProGemini 2.5 FlashClaude 3 Haiku
Source
https://medium.com/feedzaitech/benchmarking-llms-in-real-world-applications-pitfalls-and-surprises-78e720d3bfa1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

ScamAlert provides interpretable red flag detection rather than a binary score, giving users transparency about why something appears suspicious and allowing domain experts to update flag definitions as fraud tactics…

What tools did this team use?

ScamAlert, LLM, GPT-5, Claude 4, Claude 3.7, Gemini 3 Pro, Gemini 2.5 Pro, Gemini 2.5 Flash, Claude 3 Haiku.

What results were reported?

GPT-5 performance vs. competitors: significant performance over competitors; Gemini 3 Pro cost vs. GPT-5 at minimum reasoning: significantly cheaper; Claude Sonnet performance vs. similarly priced models: do not match the performance of similarly priced OpenAI or Gemini models; Latency impact of higher reasoning levels: substantial increases in latency (source-reported, not independently verified).

What failed first in this deployment?

The binary classification approach collapses nuanced scam signals into a single score, fails to account for contextual legitimacy signals the user may already hold, and provides no interpretable explanation, undermini…

How is this compliance monitoring AI workflow structured?

User submits screenshot → Multimodal model analyzes image → Response validation → Structured red flag output.

Related compliance monitoring cases