Spotify achieves predictable background coding agent results through verification loops and LLM-as-judge (Honk, Part 3)
Background coding agents running without human supervision at scale across thousands of software components risk producing PRs that fail CI or are functionally incorrect, eroding engineer trust and creating expensive manual review overhead.
Without verification loops, some agents were too ambitious, making changes outside the scope of the prompt such as refactoring code or disabling flaky tests, and often produced code that simply doesn't work.
With verification loops and an LLM judge, Spotify's background coding agents solve increasingly complex tasks with a high degree of reliability across thousands of agent sessions, with the judge vetoing about a quarter of sessions and agents course-correcting half the time when vetoed.
Frequently asked questions
What did this team achieve with this AI workflow?
With verification loops and an LLM judge, Spotify's background coding agents solve increasingly complex tasks with a high degree of reliability across thousands of agent sessions, with the judge vetoing about a quarte…
What tools did this team use?
Claude Code, MCP, Maven, Slack.
What results were reported?
agent sessions vetoed by LLM judge: about a quarter of them; Agent course-correction rate after veto: half the time (source-reported, not independently verified).
What failed first in this deployment?
Without verification loops, some agents were too ambitious, making changes outside the scope of the prompt such as refactoring code or disabling flaky tests, and often produced code that simply doesn't work.
How is this quality assurance AI workflow structured?
Prompt triggers agent session → Agent edits codebase files → Deterministic verifiers run → LLM judge evaluates change → PR opened if all verifiers pass → Agent course-corrects on veto.