Quality assurance · Production

LangChain improves coding agent 13.7 points on Terminal Bench 2.0 through harness engineering

The problem

LangChain's coding agent scored 52.8% on Terminal Bench 2.0, placing it just outside the Top 30, with identified failure modes including reasoning errors, not following task instructions, missing verification, and doom loops of repeated failed approaches.

First attempt

The most common failure pattern was that the agent wrote a solution, re-read its own code, confirmed it looked okay, and stopped without proper verification. Agents also got stuck in doom loops making small variations to the same broken approach.

Workflow diagram · grounded in source
1
Run agent, collect traces
integration
“Every agent action is stored in LangSmith. It also includes metrics like latency, token counts, and costs.”
2
Parallel trace error analysis
ai_action
“Fetch experiment traces from LangSmith Spawn parallel error analysis agents → main agent synthesizes findings + suggestions”
3
Optional human review
human_review
“A human can be pretty helpful in Step 3 (though not required) to verify and discuss proposed changes.”
4
Apply targeted harness changes
feedback_loop
“Aggregate feedback and make targeted changes to the harness.”
5
Agent self-verification pass
validation
“a PreCompletionChecklistMiddleware that intercepts the agent before it exits and reminds it to run a verification pass against the Task spec”
6
Loop detection intervention
validation
“We use a LoopDetectionMiddleware that tracks per-file edit counts via tool call hooks. It adds context like "…consider reconsidering your approach" after N edits to the same file.”
Reported outcome

By changing only the harness without modifying the underlying model, LangChain improved their coding agent from 52.8 to 66.5 on Terminal Bench 2.0, moving from outside Top 30 to Top 5.
Automated trace analysis saved hours of time in the improvement process.

Reported metrics
Terminal Bench 2.0 baseline score52.8
Terminal Bench 2.0 final score66.5
Leaderboard ranking improvementTop 30 to Top 5
Xhigh-only reasoning mode score53.9%
Show all 7 reported metrics
Terminal Bench 2.0 baseline score52.8
Terminal Bench 2.0 final score66.5
leaderboard ranking improvementTop 30 to Top 5
xhigh-only reasoning mode score53.9%
high reasoning mode score63.6%
token/time cost multiplier for max reasoningover 2x more tokens/time
trace analysis time savingssaves hours of time
Reported stack
deepagents-cliLangSmithTerminal Bench 2.0HarborDaytonagpt-5.2-codexPreCompletionChecklistMiddlewareLoopDetectionMiddleware
Source
https://blog.langchain.com/improving-deep-agents-with-harness-engineering/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By changing only the harness without modifying the underlying model, LangChain improved their coding agent from 52.8 to 66.5 on Terminal Bench 2.0, moving from outside Top 30 to Top 5.

What tools did this team use?

deepagents-cli, LangSmith, Terminal Bench 2.0, Harbor, Daytona, gpt-5.2-codex, PreCompletionChecklistMiddleware, LoopDetectionMiddleware.

What results were reported?

Terminal Bench 2.0 baseline score: 52.8; Terminal Bench 2.0 final score: 66.5; Leaderboard ranking improvement: Top 30 to Top 5; Xhigh-only reasoning mode score: 53.9% (source-reported, not independently verified).

What failed first in this deployment?

The most common failure pattern was that the agent wrote a solution, re-read its own code, confirmed it looked okay, and stopped without proper verification.

How is this quality assurance AI workflow structured?

Run agent, collect traces → Parallel trace error analysis → Optional human review → Apply targeted harness changes → Agent self-verification pass → Loop detection intervention.

Related quality assurance cases