Quality assurance · Production

Notion uses Braintrust to deploy frontier AI models within hours and keep 70 engineers aligned on evaluations

The problem

As Notion's AI grew from simple prompt chains to agentic workflows with combinatorial evaluation paths, quality problems became hard to find at scale, and existing databases began breaking under the load of large LLM traces.

First attempt

Before Braintrust, quality problems at scale went unidentified, and as AI prompts grew to hundreds of thousands of tokens, standard search was too slow to navigate massive traces.

Workflow diagram · grounded in source
1
Customer experience review
trigger
“I first started working with Braintrust on my first day at Notion. I sat down in Braintrust and looked at some of the worst experiences our customers had and tried to understand how we can be better.”
2
Regression eval run
validation
“Notion runs evals that catch regressions, and evals that measure frontier performance. When a new frontier model comes out, it might pass a regression eval at 100%, similar to the last model.”
3
Frontier eval run
ai_action
“Notion runs a frontier eval that can immediately identify what it does differently, so they can start deploying it for those specific use cases”
4
LLM-as-judge evaluation
ai_action
“they can use LLMs as a judge to ensure that Notion AI never regresses on that in the future”
5
Trace search via Brainstore
integration
“Brainstore features a search indexing infrastructure built for LLM traces, for large context, and for the type of searches that developers want to do”
6
Frontier model deployment
output
“deploying frontier models within hours of release”
7
Eval-driven iteration
feedback_loop
“Now, 80% of what the AI team does is based on evaluating from feedback and traces in Braintrust; tinkering, measuring, and understanding if they are moving in the right direction”
Reported outcome

Notion now deploys frontier AI models within hours of release, with 80% of AI team work grounded in Braintrust evaluation feedback, 70 engineers aligned on evaluation practices, and meaningful quality improvements for APAC multilingual customers.

Reported metrics
AI team work based on Braintrust eval feedback80%
Engineers kept aligned on evaluation70
Time to deploy new frontier modelswithin hours of release
multilingual quality improvement for APAC customersone of the top improvements that they've had in quality in the past year
Reported stack
BraintrustBrainstore
Source
https://www.braintrust.dev/blog/notion
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Notion now deploys frontier AI models within hours of release, with 80% of AI team work grounded in Braintrust evaluation feedback, 70 engineers aligned on evaluation practices, and meaningful quality improvements for…

What tools did this team use?

Braintrust, Brainstore.

What results were reported?

AI team work based on Braintrust eval feedback: 80%; Engineers kept aligned on evaluation: 70; Time to deploy new frontier models: within hours of release; multilingual quality improvement for APAC customers: one of the top improvements that they've had in quality in the past year (source-reported, not independently verified).

What failed first in this deployment?

Before Braintrust, quality problems at scale went unidentified, and as AI prompts grew to hundreds of thousands of tokens, standard search was too slow to navigate massive traces.

How is this quality assurance AI workflow structured?

Customer experience review → Regression eval run → Frontier eval run → LLM-as-judge evaluation → Trace search via Brainstore → Frontier model deployment → Eval-driven iteration.

Related quality assurance cases