Quality assurance · Production

Weights & Biases builds o1-based AI programming agent achieving 64.6% on SWE-Bench-Verified

The problem

Building a reliable autonomous AI programming agent required addressing o1's tendency to misorder time-sequenced events and extensive iteration over hundreds of evals to achieve consistent agent behavior.

First attempt

o1 exhibited a time-ordering failure mode: after a sequence of edits and test runs, it would claim a test still failed without having re-run the test following the most recent edit.

Workflow diagram · grounded in source
1
GitHub issue received
trigger
“It's a set of 500 github issues, docker images, and held out unit tests”
2
O1 iterative code editing
ai_action
“o1 with reasoning_mode high for all agent step and editing logic”
3
Auto-command execution
feedback_loop
“the ability to register "auto-commands" that run after every editing step”
4
Gpt4o history compression
ai_action
“a gpt4o based memory component that compresses the agent's step history”
5
Cross-check rollout selection
validation
“5 parallel rollouts for each instance, and a final "crosscheck" step for choosing the best rollout, using an o1 tie-breaker”
Reported outcome

The o1-based agent resolves 64.6% of SWE-Bench-Verified issues, tops the leaderboard, and significantly outperforms OpenAI's own published o1 result.

Reported metrics
SWE-Bench-Verified issue resolution rate64.6%
Evals run during development977
Reported stack
o1gpt4oWeavePhaseshiftEval Studiodocker
Source
https://medium.com/@shawnup/the-best-ai-programmer-from-weights-biases-04cf8127afd8
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The o1-based agent resolves 64.6% of SWE-Bench-Verified issues, tops the leaderboard, and significantly outperforms OpenAI's own published o1 result.

What tools did this team use?

o1, gpt4o, Weave, Phaseshift, Eval Studio, docker.

What results were reported?

SWE-Bench-Verified issue resolution rate: 64.6%; Evals run during development: 977 (source-reported, not independently verified).

What failed first in this deployment?

o1 exhibited a time-ordering failure mode: after a sequence of edits and test runs, it would claim a test still failed without having re-run the test following the most recent edit.

How is this quality assurance AI workflow structured?

GitHub issue received → O1 iterative code editing → Auto-command execution → Gpt4o history compression → Cross-check rollout selection.

Related quality assurance cases