Ecommerce ops · Production

trivago's explore-exploit ranking model balances known and unknown accommodation inventory

The problem

trivago must surface the 25 best accommodations from millions of options for each user search, but most inventory is never seen due to a cold-start problem, and a greedy exploitation-only baseline missed high-potential unexplored listings.

First attempt

The baseline ranking feature used only the beta-binomial posterior mean, which was too greedy and only exploited well-known accommodations. The naive exploration approach could not distinguish among accommodations with identical low exposure, and some model-based variants produced negative exploration or worse quality than the naive approach.

Workflow diagram · grounded in source
1
User search triggers ranking
trigger
“out of thousands of accommodations that match a user search, you have to select the "best" 25 to show to the user”
2
Beta-binomial baseline scoring
ai_action
“One of the features in the Ranking model is calculated using historical information on the performance of inventory over the course of a fixed set of days. The feature values distribution follows a beta-binomial model. We are currently e…”
3
Optimism-based exploration (naive)
ai_action
“Optimism about the accommodation's performance can be achieved by using the mean plus a fraction of the standard deviation(σ) of the posterior. This would favor items with low impressions since they would have larger standard deviations.…”
4
Model-based candidate scoring
ai_action
“we train a model to predict the performance of an accommodation given certain features of this accommodation. We use the historical performance of high-impression inventory to train the model and obtain a prediction function f(item featu…”
5
A/B test calibration
validation
“We ran several A/B tests and used the output of the analyzed results to recommend the next test set up”
6
Production monitoring dashboards
output
“we have created monitoring dashboards to stay on top of trends of the exploration levels in our model over time”
Reported outcome

trivago deployed an exploration mechanism in production that increases exposure of high-quality unexplored inventory at no short-term revenue cost, with no significant shift in advertiser clickshares and a tunable lambda parameter for ongoing control.

Reported metrics
User value from unexplored inventoryincrease user value by exposing a higher share of high quality unexplored inventory at no short term revenue cost
Advertiser clickshare shiftno significant shift in advertiser clickshares
Reported stack
Spark
Source
https://tech.trivago.com/post/2022-11-04-explore-exploit-dilemma-in-ranking-model
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

trivago deployed an exploration mechanism in production that increases exposure of high-quality unexplored inventory at no short-term revenue cost, with no significant shift in advertiser clickshares and a tunable lam…

What tools did this team use?

Spark.

What results were reported?

User value from unexplored inventory: increase user value by exposing a higher share of high quality unexplored inventory at no short term revenue cost; Advertiser clickshare shift: no significant shift in advertiser clickshares (source-reported, not independently verified).

What failed first in this deployment?

The baseline ranking feature used only the beta-binomial posterior mean, which was too greedy and only exploited well-known accommodations.

How is this ecommerce ops AI workflow structured?

User search triggers ranking → Beta-binomial baseline scoring → Optimism-based exploration (naive) → Model-based candidate scoring → A/B test calibration → Production monitoring dashboards.

Related ecommerce ops cases