Compliance monitoring · Production
COMPL-AI: open-source benchmarking framework to evaluate LLMs for EU AI Act compliance
The problem
EU AI Act requirements for LLMs are broad, ambiguous, and non-prescriptive, leaving model providers without concrete technical benchmarks to measurably assess their AI systems for compliance.
Workflow diagram · grounded in source
1
Select model and benchmark
trigger
“model_config and model specify the model to load and evaluate”
2
Run benchmark suite
ai_action
“an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks”
3
Store results
output
“additional results files are stored in the results_folder”
4
Generate compliance report
output
“Gemma-2-9B evaluation report showing the model evaluation across 6 EU AI Act principles and the corresponding 18 technical requirements”
5
Compare on leaderboard
validation
“COMPL-AI includes a compliance-centered leaderboard, offering a unified platform to evaluate and compare existing general-purpose models”
Reported outcome
COMPL-AI provides the first open-source benchmarking suite that translates EU AI Act requirements into measurable technical benchmarks, generates compliance reports covering 6 principles and 18 requirements, and includes a public leaderboard for comparing models.
Reported metrics
Gemma-2-9B AI disclosure accuracy99%
Llama-2 70b AI disclosure accuracy89%
Reported stack
COMPL-AIHuggingFace
Source
https://mlops.community/blog/evaluate-your-llm-for-technical-compliance-with-compl-ai
Read source ↗Frequently asked questions
What did this team achieve with this AI workflow?
COMPL-AI provides the first open-source benchmarking suite that translates EU AI Act requirements into measurable technical benchmarks, generates compliance reports covering 6 principles and 18 requirements, and inclu…
What tools did this team use?
COMPL-AI, HuggingFace.
What results were reported?
Gemma-2-9B AI disclosure accuracy: 99%; Llama-2 70b AI disclosure accuracy: 89% (source-reported, not independently verified).
How is this compliance monitoring AI workflow structured?
Select model and benchmark → Run benchmark suite → Store results → Generate compliance report → Compare on leaderboard.