Workflow · Production

How Duolingo designs structured LLM-powered conversations for Video Call with Lily

The problem

Letting an LLM converse freely with language learners produces generic, off-character, off-level responses; Duolingo needed a structured pipeline to ensure every Video Call with Lily stays at the right CEFR level, matches Lily's established personality, and has a clear conversational purpose.

First attempt

Two failure modes emerged during development: combining all instructions into one prompt overloaded the LLM and produced overly complex sentences or missing vocabulary; and without mid-call evaluation, Lily would ignore learner cues and stay on her pre-assigned topic regardless of what the learner wanted to discuss.

Workflow diagram · grounded in source
1
Learning designer writes system instructions
trigger
“Duolingo Learning Designers write the instructions that the System says to the Assistant (Lily) about how to act and what to say”
2
Conversation Prep: first question generated
ai_action
“when your Video Call is ringing, that's when the System is formulating the first question”
3
List of Facts loaded into System
integration
“before Lily begins talking, the System says "Remember this User? Here's a List of Facts: They said they have two dogs, they're studying architecture, and their favorite food is tacos."”
4
Opener greeting delivered
output
“The System tells Lily what to say first. This is almost always a greeting in the target language.”
5
Main conversation back-and-forth
ai_action
“Lily and you can then go back and forth freely through the conversation. The System has instructed Lily to react to what you say and then to continue the conversation naturally.”
6
Mid-call evaluation of learner intent
validation
“we've since added an extra check that says "Does it seem like the learner wants to lead this conversation? If yes, ignore what you originally were going to talk about."”
7
Closer triggered by program
output
“the System jumps in and whispers in Lily's ear "Psst! Say it's time to go." This prevents the call from going on forever.”
8
Post-call memory extraction
feedback_loop
“after Lily hangs up, we take the call's transcript, show it to the LLM, and ask "What important information have we learned about the User?" The information gleaned is then added to a List of Facts. The updated list becomes part of the i…”
Reported outcome

Duolingo built a structured multi-prompt pipeline for Video Call with Lily featuring separate first-question generation, persistent user memory via a List of Facts, and dynamic mid-call evaluation, enabling personalized, level-appropriate speaking practice.

Reported metrics
Call experience qualitydelight and sass—and, of course, the opportunity for speaking practice
Personalization outcomepersonalized and magical
Learner speaking confidencepractice speaking without fear
Reported stack
ChatGPTClaudeGemini
Source
https://blog.duolingo.com/ai-and-video-call/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Duolingo built a structured multi-prompt pipeline for Video Call with Lily featuring separate first-question generation, persistent user memory via a List of Facts, and dynamic mid-call evaluation, enabling personaliz…

What tools did this team use?

ChatGPT, Claude, Gemini.

What results were reported?

Call experience quality: delight and sass—and, of course, the opportunity for speaking practice; Personalization outcome: personalized and magical; Learner speaking confidence: practice speaking without fear (source-reported, not independently verified).

What failed first in this deployment?

Two failure modes emerged during development: combining all instructions into one prompt overloaded the LLM and produced overly complex sentences or missing vocabulary; and without mid-call evaluation, Lily would igno…

How is this workflow AI workflow structured?

Learning designer writes system instructions → Conversation Prep: first question generated → List of Facts loaded into System → Opener greeting delivered → Main conversation back-and-forth → Mid-call evaluation of learner intent → Closer triggered by program → Post-call memory extraction.