Quality assurance · Production

Coursera builds a structured AI evaluation framework with Braintrust to ship reliable features

The problem

Coursera lacked a formal evaluation framework for AI features, relying instead on fragmented offline jobs in spreadsheets, siloed scripts per team, and manual data reviews, which made it difficult to quickly validate AI features and push them to production with confidence.

Workflow diagram · grounded in source
1
Define success criteria
trigger
“They begin by establishing exactly what "good enough" looks like before development begins. For each AI feature, they identify specific output characteristics that matter most to their users and business goals.”
2
Build evaluation dataset
ai_action
“Their team manually reviews anonymized chatbot transcripts and human-graded assignments, paying special attention to interactions with explicit user feedback (like thumbs up/down ratings). They supplement this example data with synthetic…”
3
Heuristic and LLM-as-judge evaluation
ai_action
“Their heuristic checks provide deterministic evaluation of objective criteria, like format and response structure. For more subjective assessment, they employ LLM-as-a-judge evaluations to assess quality across multiple dimensions”
4
Online monitoring and alerting
feedback_loop
“Their online monitoring logs production traffic through evaluation scorers, tracking real-time performance against established metrics and alerting on significant deviations.”
5
Offline regression testing
validation
“Offline testing runs comprehensive evaluations on curated datasets, comparing performance across different model parameters and detecting potential regressions before deployment.”
6
Rapid prototyping in playground
validation
“their rapid prototyping process creates sample use cases in Braintrust's playground, comparing different models and testing feasibility before committing to full development”
Reported outcome

Coursera's structured evaluation framework transformed their AI development process, enabling objective validation, faster iteration, and a common quality language across teams.
Coursera Coach achieved a 90% learner satisfaction rating, and automated grading delivers grades within 1 minute of submission with approximately 45× more feedback, driving a 16.7% increase in course completions.

Reported metrics
learner satisfaction rating (Coursera Coach)90%
Time to receive grade after submissionwithin 1 minute
Feedback volume increaseapproximately 45×
Course completions within a day of peer review16.7%
Reported stack
BraintrustLLMs
Source
https://www.braintrust.dev/blog/coursera
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Coursera's structured evaluation framework transformed their AI development process, enabling objective validation, faster iteration, and a common quality language across teams.

What tools did this team use?

Braintrust, LLMs.

What results were reported?

learner satisfaction rating (Coursera Coach): 90%; Time to receive grade after submission: within 1 minute; Feedback volume increase: approximately 45×; Course completions within a day of peer review: 16.7% (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Define success criteria → Build evaluation dataset → Heuristic and LLM-as-judge evaluation → Online monitoring and alerting → Offline regression testing → Rapid prototyping in playground.

Related quality assurance cases