Quality assurance · Production

Code Droid: Factory.ai autonomous agent achieves 19.27% on SWE-bench Full

The problem

Software engineering teams face productivity bottlenecks from rote, tedious programming tasks that consume capacity and slow engineering velocity at scale.

Workflow diagram · grounded in source
1
Ticket assigned to Code Droid
trigger
“Code Droid is set up to autonomously complete tickets assigned to it”
2
Planning and task decomposition
ai_action
“Droids take high-level problems and decompose them into smaller, manageable subtasks. They translate these subtasks into an action space and then reason around optimal trajectories. We have developed several techniques that improve the D…”
3
HyperCode codebase understanding
ai_action
“We've built a system called HyperCode to construct a multi-resolution representation of a given engineering system that allows Droids to synthesize deep codebase understanding. Droids autonomously construct explicit (graph) and implicit …”
4
ByteRank context retrieval
ai_action
“We leverage the insights and data in our retrieval algorithm — ByteRank — to retrieve relevant information for a given task”
5
Multi-model patch generation
ai_action
“Code Droid can choose to generate multiple trajectories for a given task, validate them using tests (existing and self-generated), and select optimal solutions from the mix. We have found it beneficial to use a sample of different models…”
6
DroidShield pre-commit security check
validation
“DroidShield performs real-time static code analysis to detect potential security vulnerabilities, bugs, or intellectual property breaches before they are committed to code”
7
Patch output and submission
output
“Code Droid ran until it generated a complete patch, without any human assistance. The patch was then submitted for evaluation.”
Reported outcome

Code Droid achieved 19.27% on SWE-bench Full and 31.67% on SWE-bench Lite (pass@1), improving to 42.67% at pass@6, while outperforming Devin and AiderGPT4o on a comparative subset.

Reported metrics
SWE-bench Full pass rate19.27%
SWE-bench Lite pass@131.67%
SWE-bench Lite pass@237.67%
SWE-bench Lite pass@642.67%
Show all 15 reported metrics
SWE-bench Full pass rate19.27%
SWE-bench Lite pass@131.67%
SWE-bench Lite pass@237.67%
SWE-bench Lite pass@642.67%
Code Droid pass rate on 25% subset21.75%
AiderGPT4o pass rate on 25% subset18.9%
Devin pass rate on 25% subset13.86%
failure rate: target file not analyzed8%
failure rate: target file not prioritized as top-58%
failure rate: target file not selected for editing6%
average patch generation time5 to 20 minutes
maximum patch generation time136 minutes
maximum token usage per patch13 million tokens
average token usage per patchunder 2 million tokens per patch
oracle patch exact match rate1.7%
Reported stack
Code DroidHyperCodeByteRankCrucibleAnthropicOpenAI
Source
https://www.factory.ai/news/code-droid-technical-report
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Code Droid achieved 19.27% on SWE-bench Full and 31.67% on SWE-bench Lite (pass@1), improving to 42.67% at pass@6, while outperforming Devin and AiderGPT4o on a comparative subset.

What tools did this team use?

Code Droid, HyperCode, ByteRank, Crucible, Anthropic, OpenAI.

What results were reported?

SWE-bench Full pass rate: 19.27%; SWE-bench Lite pass@1: 31.67%; SWE-bench Lite pass@2: 37.67%; SWE-bench Lite pass@6: 42.67% (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Ticket assigned to Code Droid → Planning and task decomposition → HyperCode codebase understanding → ByteRank context retrieval → Multi-model patch generation → DroidShield pre-commit security check → Patch output and submission.

Related quality assurance cases