back_office_ops · workflow
Duolingo uses GPT-3 in a human-in-the-loop process to automatically generate Duolingo English Test items
Generating test content for high-stakes language proficiency tests required expert developers to manually research, ideate, and write every item — a slow, expensive process that passed costs of several hundred dollars onto test takers.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · GPT-3 generates passage
GPT-3 is used to generate a passage as the first step in creating a fill-in-the-blank test item.
Tools used
GPT-3Open AI
Outcome
By incorporating GPT-3 into a human-in-the-loop workflow, Duolingo's test developers now produce items far more efficiently from a far greater range of content, enabling a faster, more innovative test at a much more affordable price point.
Results
Cost replacedmuch more affordable price point
Grounding & classification
Source type: technical build writeup
17 fields verified against source quotes.
content generationknowledge basehuman review describednamed customerproduction runtime claimedtools describedworkflow describededucationcost reductionemployee productivitythroughput increasetechnical build writeupback office opsai draft human approval