Quality assurance · Production

LLM-as-judge evaluation framework for GenAI applications: lessons from Booking.com

The problem

Evaluating LLM-powered applications is inherently difficult because LLMs can hallucinate, fail to follow instructions, and produce outputs for which no single ground truth exists; human expert review of every generation is time-consuming and expensive to the point of being practically infeasible at scale.

Workflow diagram · grounded in source

GenAI evaluation need identified

trigger

“To maximize the potential of Generative AI (GenAI) applications and mitigate the risks associated with them, we built a framework capable of thoroughly evaluating the performance of an LLM on a specific task in a nearly automated way”

Golden dataset annotation

human_review

“The LLM-as-judge approach, requires human involvement only once (unless the production distribution changes), to carefully annotate the so-called golden dataset which must be large enough to be representative of the data distribution in …”

Judge-LLM prompting

ai_action

“we can prompt (or fine-tune) an LLM to replicate human judgement as much as possible”

Agreement threshold validation

validation

“when the agreement between the judge LLM and the human annotation has an accuracy above a certain threshold”

Evaluation report generation

output

“An evaluation report can be produced using the predictions of the target LLM and the label provided by the Judge LLM”

Production anomaly alerting

feedback_loop

“An automated system is set-up to alert the application owners whenever an anomaly is detected for any of the relevant metrics”

Reported outcome

The team built a nearly automated LLM evaluation framework using an LLM-as-judge approach, enabling continuous monitoring of GenAI application performance in production with minimal human involvement and automated anomaly alerting.

Reported metrics

Evaluation automation levelnearly automated way

Human involvement in production monitoringminimal human involvement

Reported stack

GPT-4.1Claude 4.0 SonnetDeepEval's G-EvalArize Phoenix

Source

https://mlops.community/blog/llm-evaluation-practical-tips-at-bookingcom

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The team built a nearly automated LLM evaluation framework using an LLM-as-judge approach, enabling continuous monitoring of GenAI application performance in production with minimal human involvement and automated ano…

What tools did this team use?

GPT-4.1, Claude 4.0 Sonnet, DeepEval's G-Eval, Arize Phoenix.

What results were reported?

Evaluation automation level: nearly automated way; Human involvement in production monitoring: minimal human involvement (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

GenAI evaluation need identified → Golden dataset annotation → Judge-LLM prompting → Agreement threshold validation → Evaluation report generation → Production anomaly alerting.