Quality assurance · Production

Anthropic: Infrastructure resource configuration shifts agentic coding benchmark scores by up to 6 percentage points

The problem

Infrastructure configuration alone can produce benchmark score differences exceeding the margins between frontier models on agentic coding evals; Anthropic's Kubernetes setup enforced resource specs as both floor and ceiling, causing OOM kills from transient spikes and infra error rates as high as 6% of tasks.

First attempt

Setting the guaranteed resource allocation equal to the hard kill threshold left zero headroom for transient memory spikes, causing spurious OOM kills for containers that would otherwise have succeeded.

Workflow diagram · grounded in source
1
Agent receives coding task
trigger
“models are given a full environment where they write programs, run tests, install dependencies, and iterate over multiple turns”
2
Agent attempts solution
ai_action
“The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites”
3
Container resource enforcement
validation
“each container was guaranteed the specified resources but killed the moment it exceeded them”
4
Multi-configuration experiment
output
“we ran Terminal-Bench 2.0 across six resource configurations, from strict enforcement of the per-task specs (1x), having them act as both floor and ceiling, to completely uncapped”
5
SWE-bench cross-validation
validation
“running a crossover experiment on SWE-bench. We varied the total available RAM up to 5x the baseline across 227 problems with 10 samples each”
Reported outcome

Across six resource configurations, infra error rates fell from 5.8% to 0.5% and success scores rose by 6 percentage points from strict to uncapped allocation; Anthropic recommends evals specify separate guaranteed allocation and hard kill threshold parameters per task.

Reported metrics
Tasks failing due to infra pod errors6%
Terminal-Bench score gap (most vs least resourced)6 percentage points
Infra error rate at strict enforcement (1x)5.8%
Infra error rate uncapped0.5%
Show all 8 reported metrics
tasks failing due to infra pod errors6%
Terminal-Bench score gap (most vs least resourced)6 percentage points
infra error rate at strict enforcement (1x)5.8%
infra error rate uncapped0.5%
infra error rate at 3x headroom2.1%
success rate lift from 1x to uncapped+6 percentage points
success rate jump from 3x to uncappedalmost 4 percentage points
SWE-bench score lift at 5x vs 1x RAM1.54 percentage points
Reported stack
Terminal-Bench 2.0Google Kubernetes EngineSWE-benchClaude
Source
https://www.anthropic.com/engineering/infrastructure-noise
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Across six resource configurations, infra error rates fell from 5.8% to 0.5% and success scores rose by 6 percentage points from strict to uncapped allocation; Anthropic recommends evals specify separate guaranteed al…

What tools did this team use?

Terminal-Bench 2.0, Google Kubernetes Engine, SWE-bench, Claude.

What results were reported?

Tasks failing due to infra pod errors: 6%; Terminal-Bench score gap (most vs least resourced): 6 percentage points; Infra error rate at strict enforcement (1x): 5.8%; Infra error rate uncapped: 0.5% (source-reported, not independently verified).

What failed first in this deployment?

Setting the guaranteed resource allocation equal to the hard kill threshold left zero headroom for transient memory spikes, causing spurious OOM kills for containers that would otherwise have succeeded.

How is this quality assurance AI workflow structured?

Agent receives coding task → Agent attempts solution → Container resource enforcement → Multi-configuration experiment → SWE-bench cross-validation.

Related quality assurance cases