quality_assurance · saas · workflow
Portola empowers nontechnical domain experts to ship prompt improvements 4x faster using Braintrust
Building a trustworthy AI companion required deep domain expertise in psychology, storytelling, and conversation design—nuances that could not be captured by automated evals alone. Prompt changes required coordination between subject matter experts and engineers, creating bottlenecks that slowed iteration.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Daily chat log review
Lily Doyle, their behavioral researcher, spends about an hour each day reading through chat logs in Braintrust, looking for patterns in conversation quality.
Tools used
BraintrustPlaygroundsLoop
Outcome
Nontechnical subject matter experts own the full cycle from problem identification to production deployment, resulting in a 4x improvement in iteration velocity and 4x the number of weekly prompt iterations.
Results
Time saved4x the number of weekly prompt iterations
Volume4x improvement
Grounding & classification
Source type: vendor customer story
23 fields verified against source quotes.
conversational airagvoice aichat transcriptfailure mode describedhuman review describedmetric backednamed customerproduction runtime claimedsource backedtools describedworkflow describedsoftwarecycle time reductionemployee productivityvendor customer storyquality assuranceai draft human approvalhuman review queue