Back office ops · Production

Duolingo uses GPT-3 in a human-in-the-loop process to automatically generate Duolingo English Test items

The problem

Generating test content for high-stakes language proficiency tests required expert developers to manually research, ideate, and write every item — a slow, expensive process that passed costs of several hundred dollars onto test takers.

Workflow diagram · grounded in source
1
GPT-3 generates passage
ai_action
“to create a fill-in-the-blank item, we'll first use GPT-3 to generate a passage”
2
Select sentence to remove
human_review
“then select the sentence that's the most natural to remove, so that a test taker can still understand the passage without it, and have context clues that can help them figure out what that blank sentence should say”
3
Generate wrong answer options
ai_action
“By generating a lot of texts about similar topics, we can use those other texts as sources of incorrect answers”
4
Human filter, edit, and review
human_review
“filtering, editing, and reviewing AI-generated content to produce test items that are indistinguishable from something written by actual humans”
5
Review for accuracy, fairness, bias
validation
“our teams step in to review again for accuracy, fairness, and potential bias in the material—a crucial step in any test-development process”
Reported outcome

By incorporating GPT-3 into a human-in-the-loop workflow, Duolingo's test developers now produce items far more efficiently from a far greater range of content, enabling a faster, more innovative test at a much more affordable price point.

Reported metrics
Test development efficiencyfar more efficiently
Test price pointmuch more affordable price point
Test delivery speedfaster
Content varietywide variety of interesting content
Reported stack
GPT-3Open AI
Source
https://blog.duolingo.com/test-creation-machine-learning/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By incorporating GPT-3 into a human-in-the-loop workflow, Duolingo's test developers now produce items far more efficiently from a far greater range of content, enabling a faster, more innovative test at a much more a…

What tools did this team use?

GPT-3, Open AI.

What results were reported?

Test development efficiency: far more efficiently; Test price point: much more affordable price point; Test delivery speed: faster; Content variety: wide variety of interesting content (source-reported, not independently verified).

How is this back office ops AI workflow structured?

GPT-3 generates passage → Select sentence to remove → Generate wrong answer options → Human filter, edit, and review → Review for accuracy, fairness, bias.

Related back office ops cases