Quality assurance · Production

Duolingo reduces manual regression testing by 70% with GPT Driver

The problem

Duolingo's QA team spent a substantial portion of its bandwidth on manual regression testing of weekly releases, a process that took several hours for numerous team members each week, preventing focus on higher-value work such as supporting bug fixes and testing new features.

First attempt

An initial approach of scripting specific button-tap sequences for GPT Driver led to tests quickly ballooning into large, unwieldy lists of eventualities, as Duolingo's iterative development and extensive A/B testing made rigid step-by-step automation unreliable.

Workflow diagram · grounded in source
1
Natural language test authoring
trigger
“have them type out a description of the test case they sought to fulfill, hit run, and watch the test play out in a browser in real-time”
2
GPT Driver interprets screens toward goal
ai_action
“GPT Driver would interpret each screen as presented with its end goal in mind and continue to progress until it could no longer interpret what to do with a screen or had achieved the aforementioned success criteria—this natural language …”
3
Test run recording stored
output
“GPT Driver records live test runs and stores them for easy review”
4
QA review of recordings
human_review
“our team can now quickly scrub through recordings to ensure workflows are completed as expected, a process of minutes”
Reported outcome

Duolingo reduced manual regression testing workflows by as much as 70%, cutting a process that previously took several hours for multiple QA team members each week down to a matter of minutes.

Reported metrics
Manual regression testing workflow reduction70%
Regression testing review time after automationprocess of minutes
Previous manual regression testing timeseveral hours for numerous QA Team members every week
Reported stack
GPT DriverMobileBoost
Source
https://blog.duolingo.com/reduced-regression-testing/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Duolingo reduced manual regression testing workflows by as much as 70%, cutting a process that previously took several hours for multiple QA team members each week down to a matter of minutes.

What tools did this team use?

GPT Driver, MobileBoost.

What results were reported?

Manual regression testing workflow reduction: 70%; Regression testing review time after automation: process of minutes; Previous manual regression testing time: several hours for numerous QA Team members every week (source-reported, not independently verified).

What failed first in this deployment?

An initial approach of scripting specific button-tap sequences for GPT Driver led to tests quickly ballooning into large, unwieldy lists of eventualities, as Duolingo's iterative development and extensive A/B testing…

How is this quality assurance AI workflow structured?

Natural language test authoring → GPT Driver interprets screens toward goal → Test run recording stored → QA review of recordings.

Related quality assurance cases