Quality assurance · Production

LLM Evaluation: Practical Tips at Booking.com — Lessons from One Year of Judge-LLM Development

The problem

Evaluating LLM-powered applications is difficult because generative tasks often lack a single ground truth, human expert review is too slow and expensive to scale, and LLMs risk hallucination and failure to follow instructions.

Workflow diagram · grounded in source
1
Define evaluation metric
trigger
“Define the metric definition with the business owner in the most unambiguous way possible”
2
Annotate golden dataset
human_review
“The full dataset should consist of ideally 500- 1000 examples at least. This is because part of the dataset will be used for tuning the judge-LLM (validation set) and another part as a holdout set in order to evaluate its performance”
3
Select judge-LLM backbone
ai_action
“We typically select powerful models such as GPT-4.1, or Claude 4.0 Sonnet as the backbone for our judge-LLM”
4
Iterate via error analysis
feedback_loop
“We analyze the model's mistakes on the validation set to identify recurring issues or edge cases. Based on these insights, we revise the prompt to better align it with the desired behavior. We then repeat steps 3, 4 and 5”
5
Deploy for production monitoring
output
“the judge LLM can be employed to score the predictions of a second LLM (target LLM) on other datasets”
Reported outcome

The LLM-as-judge framework enables continuous, scalable monitoring of GenAI application performance in production with minimal human involvement, and an automated prompt engineering pipeline further reduces manual development effort.

Reported metrics
Prompt engineering time per taskanywhere from one day to a full week
Production monitoring human involvementminimal human involvement
Reported stack
GPT-4.1Claude 4.0 SonnetDeepEval's G-EvalArize Phoenix
Source
https://booking.ai/llm-evaluation-practical-tips-at-booking-com-1b038a0d6662
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM-as-judge framework enables continuous, scalable monitoring of GenAI application performance in production with minimal human involvement, and an automated prompt engineering pipeline further reduces manual dev…

What tools did this team use?

GPT-4.1, Claude 4.0 Sonnet, DeepEval's G-Eval, Arize Phoenix.

What results were reported?

Prompt engineering time per task: anywhere from one day to a full week; Production monitoring human involvement: minimal human involvement (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Define evaluation metric → Annotate golden dataset → Select judge-LLM backbone → Iterate via error analysis → Deploy for production monitoring.

Related quality assurance cases