Back office ops · Production

Duolingo runs STAPLE shared task to automate generation of all acceptable translations for language course sentences

The problem

Human translators at Duolingo must manually generate every acceptable translation for each course sentence, a task that can yield enormous numbers of valid variants and significantly slows down course and content creation.

First attempt

Google Translate cannot solve this problem because it is designed to produce only a single translation per sentence, trained on generic data that has only one target translation per source.

Workflow diagram · grounded in source
1
Source sentences provided
trigger
“we provided translations from the courses that teach English to speakers of Hungarian, Japanese, Korean, Portuguese, and Vietnamese”
2
Generic MT model pre-training
ai_action
“first using millions of pairs of source and target sentences to train a generic machine translation model”
3
Fine-tune for multiple outputs
ai_action
“using our STAPLE data — with each sentence having multiple translations — to teach the model how to generate multiple correct outputs”
4
Translation set generated
output
“use computer models to produce all acceptable translations for each one – exactly the same task that our human translators currently do”
5
Correctness and completeness evaluated
validation
“We used a metric called Weighted F1 Score (based on F-measure) that gave credit to participant predictions according to a combination of correctness and completeness”
Reported outcome

Top-performing teams in the STAPLE shared task achieved Weighted F1 scores around 0.55 and dramatically outperformed Amazon Translate; Duolingo plans to use this research to build a dedicated translation tool and autocomplete suggestions for course creators.

Reported metrics
Weighted F1 Score (top team)around 0.55
performance vs Amazon Translatedramatically outperforms
Reported stack
Amazon Translate
Source
https://blog.duolingo.com/using-ai-to-open-up-bottlenecks-in-course-content-creation/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Top-performing teams in the STAPLE shared task achieved Weighted F1 scores around 0.55 and dramatically outperformed Amazon Translate; Duolingo plans to use this research to build a dedicated translation tool and auto…

What tools did this team use?

Amazon Translate.

What results were reported?

Weighted F1 Score (top team): around 0.55; performance vs Amazon Translate: dramatically outperforms (source-reported, not independently verified).

What failed first in this deployment?

Google Translate cannot solve this problem because it is designed to produce only a single translation per sentence, trained on generic data that has only one target translation per source.

How is this back office ops AI workflow structured?

Source sentences provided → Generic MT model pre-training → Fine-tune for multiple outputs → Translation set generated → Correctness and completeness evaluated.

Related back office ops cases