Quality assurance · Production

GitHub Copilot agent-driven development enables Copilot Applied Science team to ship 11 agents in under three days

The problem

Analyzing coding agent trajectories across standardized benchmark runs required reading hundreds of thousands of lines of JSON code per day—an impossible task to do manually that forced the researcher into a repetitive loop of using Copilot to surface patterns before investigating them.

Workflow diagram · grounded in source
1
Benchmark trajectory analysis trigger
trigger
“A large part of my job involves analyzing coding agent performance as measured against standardized evaluation benchmarks, like TerminalBench2 or SWEBench-Pro. This often involves poring through tons of what are called trajectories, whic…”
2
Copilot surfaces trajectory patterns
ai_action
“I used GitHub Copilot to surface patterns in the trajectories then investigated them myself—reducing the number of lines of code I had to read from hundreds of thousands to a few hundred”
3
Researcher investigates patterns
human_review
“I used GitHub Copilot to surface patterns in the trajectories then investigated them myself”
4
Plan feature with Copilot
ai_action
“Plan a new feature with Copilot using /plan. Iterate on the plan. Ensure that testing is included in the plan.”
5
Copilot implements on autopilot
ai_action
“Let Copilot implement the feature on /autopilot.”
6
Copilot Code Review agent loop
feedback_loop
“Prompt Copilot to initiate a review loop with the Copilot Code Review agent. For me, it's often something like: request Copilot Code Review, wait for the review to finish, address any relevant comments, and then re-request review. Contin…”
7
Human review and pattern enforcement
human_review
“Human review. This is where I enforce the patterns I discussed in the previous sections.”
Reported outcome

The eval-agents tool and agent-driven development methodology enabled five contributors to ship 11 new agents, four new skills, and a new concept in under three days—a change of +28,858/-2,884 lines of code across 345 files—while also reducing the lines of trajectory code the researcher had to read from hundreds of thousands to a few hundred.

Reported metrics
Trajectory lines of code requiring manual reviewreducing the number of lines of code I had to read from hundreds of thousands to a few hundred
New agents shipped11
New skills shippedfour
Time to ship 11 agents and 4 skillsless than three days
Show all 7 reported metrics
trajectory lines of code requiring manual reviewreducing the number of lines of code I had to read from hundreds of thousands to a few hundred
new agents shipped11
new skills shippedfour
time to ship 11 agents and 4 skillsless than three days
lines of code added+28,858
files changed345
new contributors onboarded to projectfive
Reported stack
GitHub CopilotCopilot CLIClaude Opus 4.6VSCodeCopilot SDKMCP serversCopilot Code Review
Source
https://github.blog/ai-and-ml/github-copilot/agent-driven-development-in-copilot-applied-science/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The eval-agents tool and agent-driven development methodology enabled five contributors to ship 11 new agents, four new skills, and a new concept in under three days—a change of +28,858/-2,884 lines of code across 345…

What tools did this team use?

GitHub Copilot, Copilot CLI, Claude Opus 4.6, VSCode, Copilot SDK, MCP servers, Copilot Code Review.

What results were reported?

Trajectory lines of code requiring manual review: reducing the number of lines of code I had to read from hundreds of thousands to a few hundred; New agents shipped: 11; New skills shipped: four; Time to ship 11 agents and 4 skills: less than three days (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Benchmark trajectory analysis trigger → Copilot surfaces trajectory patterns → Researcher investigates patterns → Plan feature with Copilot → Copilot implements on autopilot → Copilot Code Review agent loop → Human review and pattern enforcement.

Related quality assurance cases