Quality assurance · Production

Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI

The problem

Generative AI outputs are probabilistic and capable of subtle errors in tone, accuracy, and safety, making evaluation uniquely challenging. Thumbtack's early decentralized evaluation approach — where individual product teams ran their own evaluations — created duplicated effort and siloed learnings as AI features multiplied across the company.

First attempt

The initial decentralized evaluation model, in which each product team ran its own evaluations independently, became unsustainable as AI surfaces multiplied, producing duplicated effort and siloed learnings company-wide.

Workflow diagram · grounded in source
1
Curate representative datasets
trigger
“We curate datasets that reflect realistic customer and pro interactions, both with each other and with our product. These are continually expanded as new use cases appear.”
2
Codify rubrics from guidelines
validation
“Design, linguistic, and safety guidelines are distilled into evaluation rubrics. These rubrics define expectations for clarity, tone, groundedness, usefulness, and safety.”
3
Rule-based structural checks
validation
“Rule-Based Checks For structure, formatting, length, and schema validation.”
4
AI-as-a-Judge evaluation
ai_action
“AI-as-a-Judge Large language models assess clarity, tone, accuracy, relevance, and groundedness. These semantic evaluations scale human judgment.”
5
Trust and safety expert review
human_review
“Trust & Safety expert review”
6
Crowdsourced human validation
human_review
“Crowdsourced Human Review ( MTurk): Crowdsourced reviewers validate a representative sample of AI-approved content during pre-production and production.”
7
Automated retry on failure
feedback_loop
“Automated retry loops where evaluation feedback drives content regeneration”
8
Continuous monitoring and alerting
feedback_loop
“In addition, automated checks run regularly to detect regressions, drift, tone/style changes, and safety issues. Teams receive quality alerts when something shifts unexpectedly.”
Reported outcome

Thumbtack consolidated into a dedicated Evals team owning shared tooling, content guidelines, and human oversight, with the aspiration that every AI workflow ships with evaluation gates covering accuracy, trust, safety, latency, and in-product effectiveness.

Reported metrics
Human review sample rate before deployment5%
LLM Judge scoring dimensions11
Reported stack
MLflowDeepEvalMTurkDatabricksBigQuerygpt-4o-minigpt-4oPromptRefiner
Source
https://medium.com/thumbtack-engineering/evaluating-ai-at-scale-how-thumbtack-approaches-reliability-safety-and-quality-in-genai-f75d0211ac54
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Thumbtack consolidated into a dedicated Evals team owning shared tooling, content guidelines, and human oversight, with the aspiration that every AI workflow ships with evaluation gates covering accuracy, trust, safet…

What tools did this team use?

MLflow, DeepEval, MTurk, Databricks, BigQuery, gpt-4o-mini, gpt-4o, PromptRefiner.

What results were reported?

Human review sample rate before deployment: 5%; LLM Judge scoring dimensions: 11 (source-reported, not independently verified).

What failed first in this deployment?

The initial decentralized evaluation model, in which each product team ran its own evaluations independently, became unsustainable as AI surfaces multiplied, producing duplicated effort and siloed learnings company-wide.

How is this quality assurance AI workflow structured?

Curate representative datasets → Codify rubrics from guidelines → Rule-based structural checks → AI-as-a-Judge evaluation → Trust and safety expert review → Crowdsourced human validation → Automated retry on failure → Continuous monitoring and alerting.

Related quality assurance cases