Quality assurance · Production

Cursor's continuous improvement system for its AI coding agent harness

The problem

Building a reliable AI coding agent harness requires accurately measuring quality beyond benchmarks, catching tool-call degradations at scale, and customizing behavior per model — all while system complexity grows with each new model and capability added.

First attempt

Early static context-engineering guardrails were deprecated as model capabilities improved, and an experiment using a more expensive model for context summarization showed negligible quality improvement.

Workflow diagram · grounded in source
1
Vision-driven hypothesis
trigger
“Much of the work is vision-driven, where we start with an opinion about what the ideal agent experience should look like. From there, we form hypotheses about how to get closer to that vision”
2
Offline eval via CursorBench
validation
“our own eval suite, CursorBench, which gives us a fast, standardized read on quality and lets us compare across time”
3
Online A/B harness test
validation
“we also run online experiments where we deploy two or more harness variants side by side and A/B test them on real usage”
4
Keep Rate tracking
validation
“the "Keep Rate" of agent-generated code. For a given set of code changes that the agent proposed, we track what fraction of those remain in the user's codebase after fixed intervals of time”
5
LM satisfaction scoring
ai_action
“we use a language model to read the user's responses to the agent's initial output in order to capture semantically whether the user was satisfied or not. A user moving on to the next feature is a strong signal the agent did its job, whi…”
6
Error classification by cause
routing
“We classify these expected errors by cause. InvalidArguments and UnexpectedEnvironment capture model mistakes and contradictions in the context window, while ProviderError captures vendor outages from tools like GenerateImage or WebSearch”
7
Anomaly detection alerting
validation
“we have anomaly detection alerts which fire when expected errors significantly exceed the baseline. We compute baselines per-tool and per-model”
8
Weekly automated log review
ai_action
“We also run a weekly Automation equipped with a skill that teaches the model how to search through our logs, surface issues that are new or recently spiked, and create or update tickets in a backlog with an investigation”
9
Cloud Agent issue fixing
ai_action
“We lean heavily on Cloud Agents to kick off fixes for many issues at once”
Reported outcome

Over a focused sprint, Cursor drove unexpected tool call errors down by an order of magnitude and established a continuous automated loop for detecting, investigating, and fixing harness degradations.

Reported metrics
Unexpected tool call errorsdown by an order of magnitude
Agent quality improvement from expensive summarization modelnegligible difference
Reported stack
CursorBenchAutomationCloud AgentsGenerateImageWebSearchLinear
Source
https://cursor.com/blog/continually-improving-agent-harness
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Over a focused sprint, Cursor drove unexpected tool call errors down by an order of magnitude and established a continuous automated loop for detecting, investigating, and fixing harness degradations.

What tools did this team use?

CursorBench, Automation, Cloud Agents, GenerateImage, WebSearch, Linear.

What results were reported?

Unexpected tool call errors: down by an order of magnitude; Agent quality improvement from expensive summarization model: negligible difference (source-reported, not independently verified).

What failed first in this deployment?

Early static context-engineering guardrails were deprecated as model capabilities improved, and an experiment using a more expensive model for context summarization showed negligible quality improvement.

How is this quality assurance AI workflow structured?

Vision-driven hypothesis → Offline eval via CursorBench → Online A/B harness test → Keep Rate tracking → LM satisfaction scoring → Error classification by cause → Anomaly detection alerting → Weekly automated log review → Cloud Agent issue fixing.

Related quality assurance cases