Quality assurance · Production

Harvey scales AI evaluation for legal work through expert feedback, automated pipelines, and custom data infrastructure

The problem

Ensuring Harvey's AI systems consistently deliver accurate, helpful, and properly sourced legal answers requires evaluation that can scale beyond manual expert review, which is constrained by data scarcity, feedback latency, fragmented expertise across jurisdictions, and regression risks when changes improve one area but degrade another.

Workflow diagram · grounded in source
1
Domain expert review
human_review
“By collaborating with legal specialists, tax experts, and other subject matter experts, we ensure every improvement we make is grounded in real-world needs”
2
Golden retrieval dataset creation
validation
“we collaborate with domain experts to develop retrieval datasets: curated "golden" query sets designed to rigorously test how well our systems surface relevant documents. These queries range from common user questions to highly nuanced l…”
3
Retrieval system evaluation
validation
“We then evaluate our retrieval systems—both traditional and agent-based—against these references using metrics like precision (the proportion of relevant results), recall (how many of the relevant documents were found), and NDCG”
4
A/B answer preference testing
human_review
“A/B preference tests: Experts see two anonymized answers side-by-side, with model order randomized, and choose the better one”
5
Likert-scale answer rating
human_review
“Likert-scale ratings: Experts rate each answer independently on a scale from 1 (very bad) to 7 (very good), assessing dimensions like accuracy, helpfulness, and clarity”
6
Nightly canary evaluations
validation
“We run a suite of lightweight canary evaluations nightly to validate the day's code changes before they go to production, catching regressions in sourcing accuracy, answer quality, legal precision, and more”
7
LLM citation verification
ai_action
“an LLM performs a binary document-matching evaluation, confirming whether the retrieved candidate refers to the same document as the original citation”
8
Evaluation feedback to model decisions
feedback_loop
“the feedback helped validate our decision to shift more workloads to GPT-4.1, knowing it would lead to a measurable jump in quality for end users”
Reported outcome

Harvey's evaluation system validated shifting workloads to GPT-4.1, which improved mean answer ratings by over 10%, and the citation verification system achieved over 95% accuracy on an internal benchmark validated by attorneys.

Reported metrics
GPT-4.1 vs GPT-4o mean rating improvementover 10%
GPT-4.1 median score on 7-point scalefrom 5 to 6
Citation verification accuracy on internal benchmarkover 95%
Reported stack
GPT-4.1GPT-4oLLMcustom embedding pipeline
Source
https://www.harvey.ai/blog/scaling-ai-evaluation-through-expertise
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Harvey's evaluation system validated shifting workloads to GPT-4.1, which improved mean answer ratings by over 10%, and the citation verification system achieved over 95% accuracy on an internal benchmark validated by…

What tools did this team use?

GPT-4.1, GPT-4o, LLM, custom embedding pipeline.

What results were reported?

GPT-4.1 vs GPT-4o mean rating improvement: over 10%; GPT-4.1 median score on 7-point scale: from 5 to 6; Citation verification accuracy on internal benchmark: over 95% (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Domain expert review → Golden retrieval dataset creation → Retrieval system evaluation → A/B answer preference testing → Likert-scale answer rating → Nightly canary evaluations → LLM citation verification → Evaluation feedback to model decisions.

Related quality assurance cases