Quality assurance · Production
Zapier iterates AI products from sub-50% to 90%+ accuracy using Braintrust evals
The problem
AI teams commonly get stuck after shipping a v1 because there is no reliable way to know whether a prompt or code change improves overall performance or introduces regressions elsewhere.
Workflow diagram · grounded in source
1
Validate idea with frontier models
ai_action
“In this initial phase, Zapier only uses the smartest (and therefore the most expensive/slow) models - GPT-4 Turbo and Claude Opus”
2
Ship v1 to real users
output
“After validating your idea, you should build and ship v1. Sub 50% accuracy is okay!”
3
Collect explicit and implicit feedback
feedback_loop
“After shipping v1, you should obsessively collect every piece of feedback you can. This includes both explicit feedback (e.g. thumbs up/down or stars) and implicit feedback (e.g. errors, whether the user accepted a change or asked a foll…”
4
Build eval test sets in Braintrust
validation
“The Zapier team uses Braintrust to log user interactions, dig into their logs, track customer feedback, filter on that customer feedback, and directly add interesting logs to their test sets”
5
Run evals and decide to ship
feedback_loop
“After you construct a test set and run an evaluation, Braintrust also helps you understand high-level performance, dig into specific examples where your model performs poorly/well, and filter by which examples got better or worse. This g…”
6
Optimize cost and latency
validation
“the Zapier team will begin thinking through how to optimize cost and latency. By the time you reach this step, you should have a robust set of evals to test your AI product on, making it straightforward to benchmark how swapping in cheap…”
Reported outcome
Using an eval-driven feedback loop through Braintrust, Zapier improved many of their AI products from sub-50% accuracy to 90%+ within 2-3 months.
Reported metrics
AI product accuracy improvementsub-50% accuracy to 90%+
Accuracy improvement timeframe2-3 months
Reported stack
BraintrustGPT-4 TurboClaude OpusGPT-4o
Frequently asked questions
What did this team achieve with this AI workflow?
Using an eval-driven feedback loop through Braintrust, Zapier improved many of their AI products from sub-50% accuracy to 90%+ within 2-3 months.
What tools did this team use?
Braintrust, GPT-4 Turbo, Claude Opus, GPT-4o.
What results were reported?
AI product accuracy improvement: sub-50% accuracy to 90%+; Accuracy improvement timeframe: 2-3 months (source-reported, not independently verified).
How is this quality assurance AI workflow structured?
Validate idea with frontier models → Ship v1 to real users → Collect explicit and implicit feedback → Build eval test sets in Braintrust → Run evals and decide to ship → Optimize cost and latency.