Quality assurance · Production

Trae AI achieves #1 on SWE-bench Verified with 70.6% score via multi-agent patch generation and selection

The problem

Simple LLM-based patch selection degraded in performance as the candidate sampling space grew, preventing effective use of the test-time scaling law for software issue resolution.

First attempt

LLM-as-a-Selector, which used OpenAI o1 to pick among candidate patches after regression filtering, peaked at small sampling sizes and then performed worse at larger ones, undermining the benefit of generating more candidates.

Workflow diagram · grounded in source
1
Issue description triggers generation
trigger
“we leverage a Coder agent to generate candidate patches based on the issue description”
2
Multi-LLM patch generation
ai_action
“we employ multiple popular LLMs (Claude 3.7 Sonnet, Gemini 2.5 Pro, and o4 mini) as the Coder”
3
Regression test filtering
validation
“the Tester agent automatically retrieves a subset of regression tests from the original project codebase that are relevant to the given issue description. It then runs regression tests on the candidate patches and eliminates those that f…”
4
Syntax-based patch voting
ai_action
“The Selector agent first performs syntax-based voting by clustering the candidate patches based on syntactic equivalence and selecting the most frequent cluster as the potential solution... we use Tree-sitter to parse each updated code v…”
5
Dual-verification of voted patch
validation
“Leveraging both contextual information (i.e., the issue description and project codebase) and four tools (i.e., str_replace_editor, Bash, ckg_tools, and sequential_thinking_tool), the Selector agent verifies whether the syntax-voted patc…”
6
Multi-Selector agent voting
ai_action
“Each Selector agent is then tasked with selecting the most likely correct patch from the deduplicated set. The final patch is chosen by aggregating the votes from multiple Selector agents, with the patch receiving the highest number of v…”
7
Final patch returned
output
“the patch receiving the highest number of votes being returned as the final output”
Reported outcome

Trae's multi-agent Selector approach raised the overall SWE-bench Verified success rate to 70.6%, achieving the #1 position on the leaderboard when evaluated with Claude 3.7.

Reported metrics
SWE-bench Verified overall success rate70.6%
Claude-3.7-Sonnet single-attempt resolve rate60.6% to 62.6%
Gemini-2.5-Pro-0506 single-attempt resolve rate52.4% to 55%
OpenAI o4-mini single-attempt resolve rate54.4% to 55.8%
Reported stack
Tree-sitterAgentlessClaude 3.7 SonnetGemini 2.5 Proo4 miniOpenAI o1
Source
https://www.trae.ai/blog/product_update_0528
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Trae's multi-agent Selector approach raised the overall SWE-bench Verified success rate to 70.6%, achieving the #1 position on the leaderboard when evaluated with Claude 3.7.

What tools did this team use?

Tree-sitter, Agentless, Claude 3.7 Sonnet, Gemini 2.5 Pro, o4 mini, OpenAI o1.

What results were reported?

SWE-bench Verified overall success rate: 70.6%; Claude-3.7-Sonnet single-attempt resolve rate: 60.6% to 62.6%; Gemini-2.5-Pro-0506 single-attempt resolve rate: 52.4% to 55%; OpenAI o4-mini single-attempt resolve rate: 54.4% to 55.8% (source-reported, not independently verified).

What failed first in this deployment?

LLM-as-a-Selector, which used OpenAI o1 to pick among candidate patches after regression filtering, peaked at small sampling sizes and then performed worse at larger ones, undermining the benefit of generating more ca…

How is this quality assurance AI workflow structured?

Issue description triggers generation → Multi-LLM patch generation → Regression test filtering → Syntax-based patch voting → Dual-verification of voted patch → Multi-Selector agent voting → Final patch returned.

Related quality assurance cases