Workflow · Production

Duolingo designs LLM-powered AI video calls with Lily for language speaking practice

The problem

Using LLMs for language learning is not straightforward — simply instructing the model to speak a language with a learner is insufficient, and the LLM needs structured prompts with clear goals and predictable sentence structure.

First attempt

When first-question generation instructions were combined with main conversation instructions in a single prompt, the LLM became overloaded and produced undesired results — either overly complex sentences or failure to include required vocabulary.

Workflow diagram · grounded in source
1
Pre-call first question generation
ai_action
“여러분의 영상통화 벨이 울리는 와중에, 시스템은 첫 질문을 생성 중이랍니다”
2
System instructs Lily
integration
“릴리의 성격과 배경에 관한 정보, 학습자가 뭐라고 말해야 할지 모를 경우 대처법, 학습자의 레벨에 맞게 말하기 속도 유지하기 등에 관한 정보가 지시사항에 포함된답니다”
3
Lily opens the conversation
output
“시스템이 릴리에게 처음 대화 시작법을 알려줘요. 거의 모든 경우 학습 언어로 인사를 하죠. 듀오링고 엔지니어들이 각 CEFR 레벨에 맞는 다양한 인사표현을 저장해 두었답니다”
4
Mid-call learner intent evaluation
validation
“학습자가 지금 이 대화를 이끌고 싶어하는 것 같나요? 그렇다면, 말하려고 했던 대화 주제를 무시하세요”
5
Post-call memory extraction
feedback_loop
“영상통화가 하나 끝나고 나면, 그 대화 내용이 LLM에 전달돼요. 그리고 '유저에 대해 무슨 중요한 정보를 배웠나요?'라고 질문해요. 수집된 정보는 유저의 '정보 목록'에 저장돼요”
Reported outcome

Duolingo achieved more natural and appropriately leveled AI video calls by separating first-question generation from main call instructions and adding a mid-call evaluation step to detect when learners want to lead the conversation.

Reported stack
LLM
Source
https://blog.duolingo.com/ko/generative-ai-duolingo-features/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Duolingo achieved more natural and appropriately leveled AI video calls by separating first-question generation from main call instructions and adding a mid-call evaluation step to detect when learners want to lead th…

What tools did this team use?

LLM.

What failed first in this deployment?

When first-question generation instructions were combined with main conversation instructions in a single prompt, the LLM became overloaded and produced undesired results — either overly complex sentences or failure t…

How is this workflow AI workflow structured?

Pre-call first question generation → System instructs Lily → Lily opens the conversation → Mid-call learner intent evaluation → Post-call memory extraction.