Quality assurance · Production

Loom builds a repeatable AI evaluation system using Braintrust to ship AI features faster

The problem

Loom's engineering team had no systematic framework for evaluating whether AI-generated outputs like video titles were good — there was no clear, structured way to measure quality before shipping AI features.

Workflow diagram · grounded in source
1
Feature kickoff analysis
trigger
“When kicking off a new feature, their team looks at: The input: what data or prompt is the model receiving? The output: what is the model supposed to generate?”
2
Define quality traits
validation
“By identifying these traits, the team can begin to outline how they'll measure each aspect of a great output, even before writing any actual scoring code or prompts”
3
Build code-based scorers
validation
“Whenever possible, Loom automates these quality checks with deterministic, code-based scorers. Objective checks like "Does the output text contain exactly one emoji at the end?" or "Does the JSON response contain all required keys?" can …”
4
LLM-as-judge scoring
ai_action
“iterate on LLM-as-a-judge scorers with chain-of-thought rationale”
5
Initial eval iteration
feedback_loop
“feeding in around 10-15 test examples to get a feel for how the scorers are performing, inspecting the results, and refining as needed”
6
Production online evaluation
output
“they begin running evals at scale by configuring online evaluations”
Reported outcome

Loom established a repeatable evaluation system that lets them run large-scale evaluations more quickly, ship AI features with confidence, and systematically identify what works and where improvements are needed.

Reported metrics
Scorer cost and speedlow-cost and fast, even at scale
Feature shipping speed and confidenceshipping features faster and more confidently
Time and money savedsaves time and money
Reported stack
BraintrustLLMs
Source
https://www.braintrust.dev/blog/loom
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Loom established a repeatable evaluation system that lets them run large-scale evaluations more quickly, ship AI features with confidence, and systematically identify what works and where improvements are needed.

What tools did this team use?

Braintrust, LLMs.

What results were reported?

Scorer cost and speed: low-cost and fast, even at scale; Feature shipping speed and confidence: shipping features faster and more confidently; Time and money saved: saves time and money (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Feature kickoff analysis → Define quality traits → Build code-based scorers → LLM-as-judge scoring → Initial eval iteration → Production online evaluation.

Related quality assurance cases