Quality assurance · Production

Building the LLM Platform at Whatnot: Velocity, Trust, and Reliability

The problem

Whatnot needed an LLM platform capable of supporting real product and operational workflows where inputs are harder to constrain, outputs are non-deterministic, and the system is easier for users to push in unintended directions — requiring teams to iterate fast and trust results.

First attempt

Standard A/B frameworks diluted prompt experiment signal by counting all exposures regardless of whether outputs differed, and brittle rules, similarity metrics, and manual spot-checking failed to scale or capture what 'good' actually meant.

Workflow diagram · grounded in source
1
Self-serve prompt experimentation
trigger
“If someone has a better idea for how the system should behave, they should be able to test it directly in the platform instead of writing code, updating an SDK, or waiting on a deployment”
2
Differential exposure logging
validation
“we only count an exposure when the two prompt variants produce different outputs. That lets us focus measurement on the cases where the experiment could actually matter and makes prompt iteration meaningfully faster (10x+ faster)”
3
Shared tool catalog access
integration
“engineers should be able to define a tool once in Python, register it, and have it automatically show up in a shared catalog for prompt authors”
4
LLM-as-a-judge scoring
ai_action
“LLM judges help because they can score open-ended responses against a rubric in a way that is much closer to human review. Once that Judge is aligned, it becomes much more than an offline scoring tool: teams can attach it to a real produ…”
5
Data mining and eval evolution
feedback_loop
“using data mining to automatically surface weird, rare, or high-value examples for human review, so the evaluation set evolves with the product instead of freezing in time”
Reported outcome

Whatnot built an LLM platform enabling prompt iteration 10x+ faster, trust reviewers processing harassment reports in minutes instead of hours, and support agents resolving buyer issues on the first try.

Reported metrics
Prompt iteration speed10x+ faster
Trust reviewer processing timeminutes instead of hours
Reported stack
Python
Source
https://medium.com/whatnot-engineering/the-model-is-the-easy-part-building-the-llm-platform-at-whatnot-ec8730fa9bdf
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Whatnot built an LLM platform enabling prompt iteration 10x+ faster, trust reviewers processing harassment reports in minutes instead of hours, and support agents resolving buyer issues on the first try.

What tools did this team use?

Python.

What results were reported?

Prompt iteration speed: 10x+ faster; Trust reviewer processing time: minutes instead of hours (source-reported, not independently verified).

What failed first in this deployment?

Standard A/B frameworks diluted prompt experiment signal by counting all exposures regardless of whether outputs differed, and brittle rules, similarity metrics, and manual spot-checking failed to scale or capture wha…

How is this quality assurance AI workflow structured?

Self-serve prompt experimentation → Differential exposure logging → Shared tool catalog access → LLM-as-a-judge scoring → Data mining and eval evolution.

Related quality assurance cases