Compliance monitoring · Production

SumUp uses an LLM-driven evaluator to assess AML financial crime report generation

The problem

SumUp's risk and compliance agents had to manually write repetitive financial crime reports for AML escalations, and standard NLP metrics could not adequately evaluate whether LLM-generated narratives were accurate and complete.

First attempt

The Rouge Score metric showed minimal differences between accurately and inaccurately generated narratives, making it unreliable for distinguishing report quality in the AML context.

Workflow diagram · grounded in source
1
Agent confirms suspicious activity
human_review
“Once an agent confirms that an account has suspicious activity, they need to escalate and report it to the corresponding authority by writing a financial crime report.”
2
LLM generates report narrative
ai_action
“Using a combination of the agent's investigation and an LLM, documenting suspicious transactional activity is greatly optimised without the risk of an automated machine-driven false positive.”
3
LLM evaluator scores narrative
validation
“we instructed an LLM to create a score between 0–5, where 5 represents excellent coverage, and 0 signifies very poor coverage”
4
Agents validate evaluation results
human_review
“We ran an initial iteration asking the agents to manually review and assess each LLM-generated narrative with a comment and a numeric score.”
Reported outcome

The LLM-driven evaluator consistently differentiated between good and poor narratives and correlated closely with human agent assessments, enabling data scientists to test model improvements without adding extra workload to compliance agents.

Reported metrics
time savings from LLM report generationsave a substantial amount of time
LLM evaluator average score for accurately generated text4.67
LLM evaluator average score for inaccurately generated text2.5
Reported stack
LLMRouge Score
Source
https://medium.com/inside-sumup/evaluating-the-performance-of-an-llm-application-that-generates-free-text-narratives-in-the-context-c402a0136518
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM-driven evaluator consistently differentiated between good and poor narratives and correlated closely with human agent assessments, enabling data scientists to test model improvements without adding extra workl…

What tools did this team use?

LLM, Rouge Score.

What results were reported?

time savings from LLM report generation: save a substantial amount of time; LLM evaluator average score for accurately generated text: 4.67; LLM evaluator average score for inaccurately generated text: 2.5 (source-reported, not independently verified).

What failed first in this deployment?

The Rouge Score metric showed minimal differences between accurately and inaccurately generated narratives, making it unreliable for distinguishing report quality in the AML context.

How is this compliance monitoring AI workflow structured?

Agent confirms suspicious activity → LLM generates report narrative → LLM evaluator scores narrative → Agents validate evaluation results.

Related compliance monitoring cases