Finance ops · Production

DABstep: Adyen and Hugging Face benchmark multi-step reasoning for data analysis agents

The problem

Real-world data analysis requires multi-step reasoning, domain knowledge, and iterative code execution, but proper evaluation benchmarks for AI agents tackling such tasks are lacking and hinder progress in the field.

First attempt

Existing benchmarks were inadequate: DS-1000 tasks are single-shot without real datasets; DS Bench is Excel-based and uses GPT-4 as evaluator, introducing bias; benchmarks like GAIA, MATH, and SimpleQA can be answered with single-shot code generation.

Workflow diagram · grounded in source
1
Task posed to agent
trigger
“Each task contains the following items: A question that proposes a challenge to the analyst. A level encapsulating the difficulty of the task. Guidelines on how to format the answer to meet the specifications of the evaluation.”
2
Structured and unstructured data access
ai_action
“DABstep contains both unstructured and structured data to measure domain knowledge and technical skills respectively”
3
Multi-step iterative problem-solving
ai_action
“None of the tasks can be solved with 1-shot of code; in other words, they cannot be solved by reasoning alone, but rather, they require sequential steps of iterative problem-solving”
4
Factoid answer evaluation
validation
“we use adaptive tolerance to compare numerical values, allowing for variations in precision and formatting. Strings are normalized and compared using fuzzy matching with a similarity ratio threshold”
5
Real-time leaderboard submission
output
“DABstep features a real-time leaderboard powered by Hugging Face, where participants can submit their answers to be graded instantly”
Reported outcome

DABstep was released with over 450 real-world tasks from Adyen's workloads; current best-performing AI agents achieve only 16% accuracy, revealing a significant gap between current AI capability and human-level data analysis.

Reported metrics
o3-mini accuracy on DABstep16%
DeepSeek R1 accuracy on DABstep13%
Claude Sonnet accuracy on DABstep12%
DeepSeek V3 accuracy on DABstep6%
Show all 7 reported metrics
o3-mini accuracy on DABstep16%
DeepSeek R1 accuracy on DABstep13%
Claude Sonnet accuracy on DABstep12%
DeepSeek V3 accuracy on DABstep6%
human baseline accuracy on easy tasks62%
Llama 70B zero-shot accuracy on easy tasksexceed 90%
DABstep task countover 450 data analysis tasks
Reported stack
smolagentso3-miniDeepSeek R1Claude SonnetDeepSeek V3Hugging Face
Source
https://medium.com/adyen/data-agent-benchmark-for-multi-step-reasoning-dabstep-70e913c339dc
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

DABstep was released with over 450 real-world tasks from Adyen's workloads; current best-performing AI agents achieve only 16% accuracy, revealing a significant gap between current AI capability and human-level data a…

What tools did this team use?

smolagents, o3-mini, DeepSeek R1, Claude Sonnet, DeepSeek V3, Hugging Face.

What results were reported?

o3-mini accuracy on DABstep: 16%; DeepSeek R1 accuracy on DABstep: 13%; Claude Sonnet accuracy on DABstep: 12%; DeepSeek V3 accuracy on DABstep: 6% (source-reported, not independently verified).

What failed first in this deployment?

Existing benchmarks were inadequate: DS-1000 tasks are single-shot without real datasets; DS Bench is Excel-based and uses GPT-4 as evaluator, introducing bias; benchmarks like GAIA, MATH, and SimpleQA can be answered…

How is this finance ops AI workflow structured?

Task posed to agent → Structured and unstructured data access → Multi-step iterative problem-solving → Factoid answer evaluation → Real-time leaderboard submission.

Related finance ops cases