Quality assurance · Production

Case Study: LLM-Generated Test Cases Achieve Comparable Quality to Manual QA on Da.tes Platform

The problem

The practical implications of integrating LLMs into test case construction for real-world software applications remained underexplored, leaving software practitioners without concrete guidance on efficacy, challenges, and trade-offs.

Workflow diagram · grounded in source

Engineer fills description template

trigger

“it was necessary to fill one template per feature in the software”

LLM generates requirements document

ai_action

“the model is guided to generate a requirements document”

LLM generates test conditions

ai_action

“the model should return the results in JSON format, containing the Requirements grouped by functional or non-functional followed by the Test Conditions of each one”

LLM generates structured test cases

output

“the model role and an output format as a structured test case, containing title, preconditions, steps, expected results, test data and test classification”

QA Engineers evaluate via forms

human_review

“create 2 Google Forms with questions evaluating 10 AI generated test cases along with 10 test cases manually written by a QA team and share with 10 QA Engineers”

Reported outcome

AI-generated test cases scored an average of 4.31 versus 4.18 for human-generated cases, and 58.6% of A/B preferences favoured AI, with the study concluding LLM-assisted test case construction produces artifacts of comparable quality to those developed manually.

Reported metrics

AI test case average quality score4.31

Human test case average quality score4.18

A/B preference for AI-generated test cases58.6%

Reported stack

GPT-3.5 TurboLangChainOpenAI API

Source

https://arxiv.org/html/2312.12598v2

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

AI-generated test cases scored an average of 4.31 versus 4.18 for human-generated cases, and 58.6% of A/B preferences favoured AI, with the study concluding LLM-assisted test case construction produces artifacts of co…

What tools did this team use?

GPT-3.5 Turbo, LangChain, OpenAI API.

What results were reported?

AI test case average quality score: 4.31; Human test case average quality score: 4.18; A/B preference for AI-generated test cases: 58.6% (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Engineer fills description template → LLM generates requirements document → LLM generates test conditions → LLM generates structured test cases → QA Engineers evaluate via forms.