Back office ops · Production

Stripe builds a benchmark to evaluate AI agents on real Stripe integrations

The problem

There was no established way to measure whether AI agents could autonomously complete long-horizon, end-to-end Stripe integrations, given the unquantified gap between LLM coding capability and the ability to manage full software engineering projects requiring planning, persistent state management, and failure recovery.

First attempt

Agents mishandled ambiguous situations by treating invalid API error responses as successful completions, and were occasionally unable to recover from browser interaction failures that a human engineer could have resolved trivially.

Workflow diagram · grounded in source

Task environment provided to agent

trigger

“A full coding environment with code, databases, and scripts that represent a typical starting repository for a Stripe integration project. This environment also includes test Stripe API keys that the agent could use for testing and chall…”

Agent builds integration using tools

ai_action

“we used a goose-based harness for all evaluation runs, and provided models with a Model Context Protocol (MCP) server that grants access to a terminal, browser, and Stripe-specific search tools”

Agent self-verifies in browser

validation

“they were asked to self-verify the change by completing a test purchase in the browser”

Automated graders score submission

validation

“graders were implemented as deterministic tests that exercised the finished software via API calls or automated UI tests, or both. Some graders also validated the Stripe artifacts of a run by inspecting created Stripe API objects”

Benchmark findings published

output

“We're sharing these benchmarks to help the broader software community advance agentic tooling.”

Reported outcome

Claude Opus 4.5 achieved a 92% average score across four full-stack tasks, GPT-5.2 achieved 73% across two gym problem sets, and best-performing runs averaged 63 turns; agents navigated UIs, debugged live issues, and handled underdocumented API behavior.

Reported metrics

Claude Opus 4.5 full-stack task average score92%

GPT-5.2 gym problem set average score73%

Best-performing run average turns63 turns

Checkout gym task correct parametersover 80%

Reported stack

gooseClaude Opus 4.5GPT-5.2AnthropicVercel

Source

https://stripe.com/blog/can-ai-agents-build-real-stripe-integrations

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Claude Opus 4.5 achieved a 92% average score across four full-stack tasks, GPT-5.2 achieved 73% across two gym problem sets, and best-performing runs averaged 63 turns; agents navigated UIs, debugged live issues, and…

What tools did this team use?

goose, Claude Opus 4.5, GPT-5.2, Anthropic, Vercel.

What results were reported?

Claude Opus 4.5 full-stack task average score: 92%; GPT-5.2 gym problem set average score: 73%; Best-performing run average turns: 63 turns; Checkout gym task correct parameters: over 80% (source-reported, not independently verified).

What failed first in this deployment?

How is this back office ops AI workflow structured?

Task environment provided to agent → Agent builds integration using tools → Agent self-verifies in browser → Automated graders score submission → Benchmark findings published.