Back office ops · Production

QualIT: LLM-enhanced topic modeling for qualitative text analysis

The problem

Manual analysis of large volumes of open-ended qualitative text is prohibitively labor-intensive, and standard topic-modeling approaches like LDA fail to capture contextual nuances and ambiguities in natural language.

First attempt

LDA and BERTopic underperformed on both topic coherence and topic diversity benchmarks, and human reviewers could match ground-truth categories less reliably with those methods.

Workflow diagram · grounded in source
1
Qualitative text ingestion
trigger
“Whether collected through employee surveys, product feedback channels, voice-of-customer mechanisms, or other unstructured text sources”
2
LLM key-phrase extraction
ai_action
“The LLM analyzes each document, identifying key phrases that capture the most salient themes and topics. This is a crucial advantage over alternative methods that characterize each document according to a single topic. By extracting mult…”
3
Hallucination check
validation
“QualIT calculates a coherence score for each one. This score assesses how well the key phrase aligns with the actual text, serving as a metric for consistency and relevance. Key phrases that fall below a certain coherence threshold are f…”
4
Two-stage hierarchical clustering
ai_action
“First, the model groups key phrases extracted by the LLM into primary clusters, representing the overarching themes present in the corpus. It then applies a secondary round of clustering within each primary cluster to identify more speci…”
5
Topic representation output
output
“QualIT is able to enrich the topic-modeling process, generating more nuanced and interpretable topic representations from free-text data”
Reported outcome

QualIT achieved 70% topic coherence and 95.5% topic diversity on the 20 Newsgroups benchmark, outperforming LDA and BERTopic on both metrics, and human reviewers matched ground truth at 50% with QualIT versus 25% for LDA and BERTopic.

Reported metrics
topic coherence — QualIT70%
topic coherence — LDA baseline65%
topic coherence — BERTopic baseline57%
topic diversity — QualIT95.5%
Show all 8 reported metrics
topic coherence — QualIT70%
topic coherence — LDA baseline65%
topic coherence — BERTopic baseline57%
topic diversity — QualIT95.5%
topic diversity — LDA baseline85%
topic diversity — BERTopic baseline72%
human reviewer ground-truth overlap — QualIT50%
human reviewer ground-truth overlap — LDA and BERTopic25%
Reported stack
QualITLLMsLDABERTopic
Source
https://www.amazon.science/blog/unlocking-insights-from-qualitative-text-with-llm-enhanced-topic-modeling?utm_campaign=unlocking-insights-from-qualitative-text&utm_term=2024-dec&utm_medium=organic-asw&utm_content=2024-12-11-unlocking-insights-from-qualitative-text&utm_source=twitter&tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

QualIT achieved 70% topic coherence and 95.5% topic diversity on the 20 Newsgroups benchmark, outperforming LDA and BERTopic on both metrics, and human reviewers matched ground truth at 50% with QualIT versus 25% for…

What tools did this team use?

QualIT, LLMs, LDA, BERTopic.

What results were reported?

topic coherence — QualIT: 70%; topic coherence — LDA baseline: 65%; topic coherence — BERTopic baseline: 57%; topic diversity — QualIT: 95.5% (source-reported, not independently verified).

What failed first in this deployment?

LDA and BERTopic underperformed on both topic coherence and topic diversity benchmarks, and human reviewers could match ground-truth categories less reliably with those methods.

How is this back office ops AI workflow structured?

Qualitative text ingestion → LLM key-phrase extraction → Hallucination check → Two-stage hierarchical clustering → Topic representation output.

Related back office ops cases