Duolingo uses machine learning to prioritize learner-submitted course fix reports
About 90% of learner-submitted translation fix reports contain errors, making it difficult for contributors to efficiently identify the roughly 10% that are valid and require action.
The previous approach of sorting reports by how often each answer was submitted (wisdom of the crowd) performed only slightly above random, achieving an AUC of about 0.59.
The ML ranking system outperformed the frequency-based approach on every single course, including low-resource languages, and enabled new courses such as Arabic, Latin, and Scottish Gaelic to reach stable low problem-report rates within weeks of launch.
Frequently asked questions
What did this team achieve with this AI workflow?
The ML ranking system outperformed the frequency-based approach on every single course, including low-resource languages, and enabled new courses such as Arabic, Latin, and Scottish Gaelic to reach stable low problem-…
What tools did this team use?
logistic regression, Duolingo Incubator.
What results were reported?
Valid reports requiring a fix: 10%; Reports containing mistakes: 90%; previous frequency-based system AUC: 0.59; ML system vs previous strategy: did better than the previous strategy on every single course (source-reported, not independently verified).
What failed first in this deployment?
The previous approach of sorting reports by how often each answer was submitted (wisdom of the crowd) performed only slightly above random, achieving an AUC of about 0.59.
How is this quality assurance AI workflow structured?
Learner submits report → ML scoring of report → Reports ranked for human review → Staff and contributor review → Accept/reject feeds training.