Quality assurance · Production

Eval-driven development: Vercel's approach to building AI products like v0

The problem

Traditional software testing methods do not work for AI's probabilistic, non-deterministic outputs, and existing eval management approaches are ad-hoc, unscalable, and lack the specificity needed to guide targeted improvement.

Workflow diagram · grounded in source
1
PR triggers eval suite
trigger
“Every GitHub pull request that impacts the output pipeline includes eval results”
2
Code-based grading
validation
“Validating code blocks Ensuring correct imports Confirming multi-file usage Verifying the balance of code comments versus actual code (correcting LLM laziness)”
3
Human feedback collection
human_review
“end user and internal human feedback”
4
LLM-based grading
ai_action
“LLM-based grading for complex judgments at scale”
5
Eval results reported
output
“An automated script runs the entire eval test suite and reports pass/fail rates, regressions, and improvements. Braintrust logs everything for manual review.”
6
Feedback feeds evals
feedback_loop
“All this feedback pours back into evals”
Reported outcome

Vercel's v0 AI product uses eval-driven development to catch errors early, speed up iteration, and maintain a 100% pass rate on refusal and safety evaluations, with prompts iterated on almost daily.

Reported metrics
Refusal and safety eval pass rate100%
LLM eval cost vs code-based grading1.5x to 2x more than code-based grading
Reported stack
BraintrustAI SDKv0GitHub
Source
https://vercel.com/blog/eval-driven-development-build-better-ai-faster?utm_source=chatgpt.com
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Vercel's v0 AI product uses eval-driven development to catch errors early, speed up iteration, and maintain a 100% pass rate on refusal and safety evaluations, with prompts iterated on almost daily.

What tools did this team use?

Braintrust, AI SDK, v0, GitHub.

What results were reported?

Refusal and safety eval pass rate: 100%; LLM eval cost vs code-based grading: 1.5x to 2x more than code-based grading (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

PR triggers eval suite → Code-based grading → Human feedback collection → LLM-based grading → Eval results reported → Feedback feeds evals.

Related quality assurance cases