Quality assurance · Production

LLM-based agents for automating the enhancement of user story quality at Austrian Post Group IT: An early report

The problem

Agile teams at Austrian Post Group IT struggled to maintain high-quality user stories at scale; existing NLP-based quality tools were limited in scope, and user stories were criticized for ambiguity and missing detail in acceptance criteria.

Workflow diagram · grounded in source
1
Task initiation
trigger
“The task initiates the interaction and contains inputs that define the work scope and objectives.”
2
Agent PO product alignment
ai_action
“Agent PO understands the vision of the project. It is responsible for managing product backlog and prioritizing user stories based on business value and customer needs. This agent ensures that the user stories align with the overall prod…”
3
Agent RE quality improvement
ai_action
“Agent RE is tailored to focus on the quality aspects of user stories. It ensures that the user story description is unambiguous, and the acceptance criteria are measurable.”
4
Shared knowledge base iteration
ai_action
“Agents, using the shared knowledge base, engage in a collaborative conversation, each contributing their expertise towards the task completion.”
5
Plan review by agile practitioners
human_review
“This plan was further reviewed and refined by a Scrum master and a PO in agile teams, ensuring that it aligns with the company's agile framework, common practice for requirements management, and project objectives.”
6
Improved user stories output
output
“The final output is an accumulation of these collaborative efforts, embodying the collective intelligence of the participating agents.”
Reported outcome

Preliminary assessment by practitioners across agile teams at Austrian Post Group IT indicated that ALAS demonstrated the potential of LLMs in improving user story quality, providing a practical example of the transformative impact of AI in an industry setting.

Reported metrics
LLM potential for user story quality improvementdemonstrate the potential of LLMs in improving user story quality
Reported stack
gpt-3.5-turbo-16kgpt-4-1106-preview
Source
https://arxiv.org/html/2403.09442v1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Preliminary assessment by practitioners across agile teams at Austrian Post Group IT indicated that ALAS demonstrated the potential of LLMs in improving user story quality, providing a practical example of the transfo…

What tools did this team use?

gpt-3.5-turbo-16k, gpt-4-1106-preview.

What results were reported?

LLM potential for user story quality improvement: demonstrate the potential of LLMs in improving user story quality (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Task initiation → Agent PO product alignment → Agent RE quality improvement → Shared knowledge base iteration → Plan review by agile practitioners → Improved user stories output.

Related quality assurance cases