Marketing ops · Production

Duolingo builds bandit algorithm to personalize daily practice reminder notifications

The problem

Duolingo's practice reminders were selected at random from a pool, missing the opportunity to tailor messages to individual learners and potentially leaving engagement gains on the table.

First attempt

Standard bandit algorithms reuse the top-performing notification repeatedly, which conflicted with a confirmed novelty effect: learners are most receptive to notifications they have not recently seen, and repetition erodes effectiveness.

Workflow diagram · grounded in source
1
A/B test new templates
validation
“we always run experiments to test new notifications on a small number of learners before using them across the board. This way, only the best templates get permanently added to the pool”
2
Bandit selects notification
ai_action
“Bandit algorithms are a form of AI where an algorithm must repeatedly choose between the same set of options, and it gradually learns from past decisions which options are best—that is, which of our notifications are most likely to get a…”
3
Novelty filter applied
validation
“we had to explicitly teach the AI algorithm that learners don't like seeing the same notification too often by demoting reminders that have already been seen recently. We decided how much to space apart the repetitions of a notification …”
4
Practice reminder sent
output
“Duolingo helps learners stay on track by sending daily practice reminders”
5
Lesson completion tracked
feedback_loop
“its "payout" is getting a learner to complete a lesson”
6
Insights improve future design
feedback_loop
“we could use the insights that our AI had learned to design better notifications in the future, so that we can boost learner motivation even more”
Reported outcome

Within weeks of deployment, more learners were completing lessons more frequently, and the system was especially successful at helping tens of thousands of new learners return to their lessons.

Reported metrics
Practice reminders in training dataset~200 million
Weekly records produced by systemtens of millions of records per week
Learner lesson completion frequencymore learners were completing lessons more frequently
New learners returning to lessonstens of thousands of new learners
Reported stack
AWS Kinesis FirehoseSpark
Source
https://blog.duolingo.com/hi-its-duo-the-ai-behind-the-meme/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Within weeks of deployment, more learners were completing lessons more frequently, and the system was especially successful at helping tens of thousands of new learners return to their lessons.

What tools did this team use?

AWS Kinesis Firehose, Spark.

What results were reported?

Practice reminders in training dataset: ~200 million; Weekly records produced by system: tens of millions of records per week; Learner lesson completion frequency: more learners were completing lessons more frequently; New learners returning to lessons: tens of thousands of new learners (source-reported, not independently verified).

What failed first in this deployment?

Standard bandit algorithms reuse the top-performing notification repeatedly, which conflicted with a confirmed novelty effect: learners are most receptive to notifications they have not recently seen, and repetition e…

How is this marketing ops AI workflow structured?

A/B test new templates → Bandit selects notification → Novelty filter applied → Practice reminder sent → Lesson completion tracked → Insights improve future design.

Related marketing ops cases