Quality assurance · Production

Portola empowers nontechnical domain experts to ship prompt improvements 4x faster using Braintrust

The problem

Building a trustworthy AI companion required deep domain expertise in psychology, storytelling, and conversation design—nuances that could not be captured by automated evals alone. Prompt changes required coordination between subject matter experts and engineers, creating bottlenecks that slowed iteration.

Workflow diagram · grounded in source
1
Daily chat log review
human_review
“Lily Doyle, their behavioral researcher, spends about an hour each day reading through chat logs in Braintrust, looking for patterns in conversation quality. "I look for recurring patterns in form and function. That means both how the me…”
2
Problem-specific dataset creation
integration
“she creates a dataset in Braintrust tagged with the specific issue. Each dataset becomes a collection of real conversation examples that demonstrate a particular problem.”
3
Side-by-side prompt comparison
human_review
“Lily moves to playgrounds for side-by-side prompt comparison. She manually reviews outputs from the current prompt versus iterations, assessing conversation quality through her domain expertise.”
4
Direct production deployment
output
“The final piece of Portola's workflow is their prompts-as-code infrastructure, which enables subject matter experts to deploy changes directly to production once they're satisfied with playground results.”
5
Weekly error pattern synthesis
feedback_loop
“Loop was our way of getting data or synthesizing log data more efficiently at an aggregate level. We use it to find common error patterns every single week.”
Reported outcome

Nontechnical subject matter experts own the full cycle from problem identification to production deployment, resulting in a 4x improvement in iteration velocity and 4x the number of weekly prompt iterations.

Reported metrics
Weekly prompt iterations4x the number of weekly prompt iterations
Iteration velocity4x improvement
Reported stack
BraintrustPlaygroundsLoop
Source
https://www.braintrust.dev/blog/portola
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Nontechnical subject matter experts own the full cycle from problem identification to production deployment, resulting in a 4x improvement in iteration velocity and 4x the number of weekly prompt iterations.

What tools did this team use?

Braintrust, Playgrounds, Loop.

What results were reported?

Weekly prompt iterations: 4x the number of weekly prompt iterations; Iteration velocity: 4x improvement (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Daily chat log review → Problem-specific dataset creation → Side-by-side prompt comparison → Direct production deployment → Weekly error pattern synthesis.

Related quality assurance cases