Quality assurance · Production

AI Agent Evaluation: Practical Tips at Booking.com

The problem

LLM agents require a more complex evaluation process than single LLMs because they use external tools, iterate through intermediate steps, and make autonomous decisions — none of which standard prompt-response evaluation captures.

Workflow diagram · grounded in source
1
User-agent chat session
trigger
“This metric is computed for each chat session between the user and the agent where the user might ask the agent to perform several tasks”
2
Judge LLM task scoring
ai_action
“The response of the judge LLM is a list of binary scores for each of the task requests identified by the judge LLM in the whole dialogue. The score per dialogue is given by the average of the values within this list.”
3
Tool call validity check
validation
“In our evaluation pipeline we validate the JSON instance with the tool call arguments with the JSON schema of the tool.”
4
Tool correctness assessment
validation
“If the set of tools used by the agent to solve a certain task corresponds to the set of expected tools we conclude that the tool calls are correct. Otherwise we throw an error and return which tools are redundant (i.e. used by the agent …”
5
Tool reliability judge checks
ai_action
“we introduce a set of simple judge LLMs that assess the quality of a tool specification. Each judge LLM performs a "Tool Reliability" check by taking the tuple (name, description, input schema) and returning a binary score indicating whe…”
6
Baseline benchmarking decision
validation
“The decision to deploy an agent should be data driven, by benchmarking the agent against one or more baselines on the following metrics: Performance, in terms of task completion or other judge LLM based metrics which can ideally translat…”
Reported outcome

Booking.com developed a dual evaluation framework combining black box task completion scoring via judge LLMs and glass box tool proficiency and reliability checks, enabling data-driven deployment decisions that weigh performance uplift against increased cost and latency.

Reported metrics
Agent consistency pass^k at k=8 trialsbelow 20%
Agent performance impact of poor tool namesdecreases significantly
Agent performance impact of improved tool descriptionssignificant performance improvements
Reported stack
judge LLMJSON schemaBooking AI Trip Planner
Source
https://booking.ai/ai-agent-evaluation-82e781439d97?source=rss----4d265f07defc---4
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Booking.com developed a dual evaluation framework combining black box task completion scoring via judge LLMs and glass box tool proficiency and reliability checks, enabling data-driven deployment decisions that weigh…

What tools did this team use?

judge LLM, JSON schema, Booking AI Trip Planner.

What results were reported?

Agent consistency pass^k at k=8 trials: below 20%; Agent performance impact of poor tool names: decreases significantly; Agent performance impact of improved tool descriptions: significant performance improvements (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

User-agent chat session → Judge LLM task scoring → Tool call validity check → Tool correctness assessment → Tool reliability judge checks → Baseline benchmarking decision.

Related quality assurance cases