Marketing ops · Production

Duolingo replaces rule-based ad logic with BigQuery ML and XGBoost, driving tens of millions in annual revenue

The problem

Duolingo's ad decision logic had become too complex to optimize after years of A/B testing created a tangled web of rules fragmented across systems, making the codebase difficult to reason about, improve, or maintain without introducing bugs and tech debt.

First attempt

Two early ML iteration problems emerged: a dedicated holdout group for retraining introduced data drift by training on a different learner population than the one served by the model, and the initial model objective of predicting baseline subscription probability cannibalized revenue from other channels rather than growing it.

Workflow diagram · grounded in source
1
User completes learning session
trigger
“After each learning session, users either see an "in-house" ad promoting our Duolingo subscription or a "network" ad that generates ad revenue from external advertisers”
2
Dbt processes fresh data
integration
“dbt already powers all of Duolingo's data pipelines, so spinning up a new job to continuously process fresh batches of data was simple”
3
BigQuery ML trains and evaluates model
ai_action
“BigQuery ML then let us use that data to train, evaluate, and analyze off-the-shelf models with just a few lines of code, all while handling hyperparameter tuning”
4
SQL inference decides ad type
ai_action
“Performing inference was then as simple as running a SQL query”
5
Decision stored in prod database
integration
“additional engineering to store model decisions in our prod database”
6
Random exploration for retraining
feedback_loop
“we occasionally ignore the model decision and serve an ad randomly, which is simpler and more effective”
7
XGBoost contextual bandit uplift
ai_action
“we turned XGBoost into a contextual bandit by identifying learners for whom showing an in-house ad increased their purchase probability the most”
Reported outcome

The ML model delivered millions of dollars in incremental annual revenue in its first few months, grew to tens of millions per year after refinement, and now drives roughly a quarter of Duolingo's year-over-year revenue growth.

Reported metrics
Incremental annual revenue (initial months)millions of dollars in incremental annual revenue
Incremental annual revenue (after refinement)tens of millions per year
Share of year-over-year revenue growthroughly a quarter of our year-over-year revenue growth
Daily active user growthsubstantial daily active user (DAU) wins
Reported stack
dbtBigQuery MLXGBoostdbt_ml
Source
https://blog.duolingo.com/machine-learning-ads/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The ML model delivered millions of dollars in incremental annual revenue in its first few months, grew to tens of millions per year after refinement, and now drives roughly a quarter of Duolingo's year-over-year reven…

What tools did this team use?

dbt, BigQuery ML, XGBoost, dbt_ml.

What results were reported?

Incremental annual revenue (initial months): millions of dollars in incremental annual revenue; Incremental annual revenue (after refinement): tens of millions per year; Share of year-over-year revenue growth: roughly a quarter of our year-over-year revenue growth; Daily active user growth: substantial daily active user (DAU) wins (source-reported, not independently verified).

What failed first in this deployment?

Two early ML iteration problems emerged: a dedicated holdout group for retraining introduced data drift by training on a different learner population than the one served by the model, and the initial model objective o…

How is this marketing ops AI workflow structured?

User completes learning session → Dbt processes fresh data → BigQuery ML trains and evaluates model → SQL inference decides ad type → Decision stored in prod database → Random exploration for retraining → XGBoost contextual bandit uplift.

Related marketing ops cases