Quality assurance · Production

Databricks coSTAR: automated AI agent testing reduces review cycle from two-week manual reviews to hours

The problem

Databricks' AI agent development relied on a slow, manual review-and-fix loop with no comprehensive automated test suite, making it impossible to iterate on agents with confidence as they grew in complexity and scope.

First attempt

The manual review loop failed predictably: without systematic tests, agents could regress silently, and manually QA-ing every change was unsustainable.

Workflow diagram · grounded in source
1
Scenario definition
trigger
“a scenario definition: a structured description of the initial state, the user prompt, and the expected outcomes”
2
Agent trace capture
integration
“To run our test suite, we use a harness that sends each scenario's prompt to the agent under test (AUT). Each execution is captured as a MLflow trace: a structured log of every tool call, every intermediate output, and every artifact the…”
3
Agentic judge scoring
validation
“we use MLflow's agentic judges: judges that are themselves agents, equipped with tools to explore the trace selectively”
4
Golden Set human review
human_review
“we first curate a Golden Set of typically dozens of examples of agent outputs that our engineers have manually assessed”
5
Judge alignment refinement
feedback_loop
“we leverage MLflow's alignment capabilities (powered by techniques like GEPA and MemAlign) to automatically refine the judge against the Golden Set”
6
Coding assistant agent refinement
ai_action
“A coding assistant treats the agent as its codebase and the judges as its test suite. It reads failures, diagnoses root causes, patches the agent, and re-runs everything. The engineer is still the reviewer and final arbiter of the propos…”
7
Production traffic monitoring
feedback_loop
“we run them (or a sampled subset) on real production conversations”
Reported outcome

Databricks moved from two-week manual reviews to automated test-and-refine in hours, adopting coSTAR across multiple production agents with tangible benefits, including automated regression detection and saved human effort.

Reported metrics
Review cycle time (before)two-week manual reviews
Review cycle time (after)automated test-and-refine in hours
Overall benefitstangible benefits
Human effort in agent analysis and improvementsaves considerable human effort
Reported stack
MLflowcoSTARGEPAMemAlignMCP tools
Source
https://www.databricks.com/blog/costar-how-we-ship-ai-agents-databricks-fast-without-breaking-things
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Databricks moved from two-week manual reviews to automated test-and-refine in hours, adopting coSTAR across multiple production agents with tangible benefits, including automated regression detection and saved human e…

What tools did this team use?

MLflow, coSTAR, GEPA, MemAlign, MCP tools.

What results were reported?

Review cycle time (before): two-week manual reviews; Review cycle time (after): automated test-and-refine in hours; Overall benefits: tangible benefits; Human effort in agent analysis and improvement: saves considerable human effort (source-reported, not independently verified).

What failed first in this deployment?

The manual review loop failed predictably: without systematic tests, agents could regress silently, and manually QA-ing every change was unsustainable.

How is this quality assurance AI workflow structured?

Scenario definition → Agent trace capture → Agentic judge scoring → Golden Set human review → Judge alignment refinement → Coding assistant agent refinement → Production traffic monitoring.

Related quality assurance cases