Workflow · Production

Supervised ML classifier detects SAP Community feature requests with 0.819 accuracy

The problem

Enterprise software vendors cannot rely on app store reviews for requirements elicitation and have no direct access to end users, while traditional techniques such as user interviews are too labor-intensive to scale across the millions of posts in developer communities.

Workflow diagram · grounded in source
1
SAP Community data collection
trigger
“they collected data from the SAP Community and generated a manually labeled data set of 1,500 questions”
2
Expert labeling via Labelbox
human_review
“The researchers used Labelbox to collect the assessments from their labelers. They used a managed-labeler approach to label our final sample of 1,500 questions. A key reason for this approach was that the assessment of whether a question…”
3
SBERT feature extraction
ai_action
“extracted features with the pre-trained SBERT-Model”
4
Naïve Bayes binary classification
ai_action
“classified them with the Naïve Bayes algorithm”
5
Accuracy validation
validation
“They observed the highest prediction accuracy (0.8187) for the classifier”
Reported outcome

The supervised binary ML classifier reached a high accuracy of 0.819, demonstrating that supervised machine learning models are an effective means for automatically identifying feature requests in developer community posts.

Reported metrics
Classifier accuracy0.819
highest prediction accuracy (SBERT + Naïve Bayes)0.8187
Labeled dataset size1,500 questions
Reported stack
LabelboxSBERT-ModelNaïve BayesFiverrSAP Community
Source
https://labelbox.com/research/detecting-feature-requests-of-third-party-developers-through-machine-learning-a-case-study-of-the-sap-community/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The supervised binary ML classifier reached a high accuracy of 0.819, demonstrating that supervised machine learning models are an effective means for automatically identifying feature requests in developer community…

What tools did this team use?

Labelbox, SBERT-Model, Naïve Bayes, Fiverr, SAP Community.

What results were reported?

Classifier accuracy: 0.819; highest prediction accuracy (SBERT + Naïve Bayes): 0.8187; Labeled dataset size: 1,500 questions (source-reported, not independently verified).

How is this workflow AI workflow structured?

SAP Community data collection → Expert labeling via Labelbox → SBERT feature extraction → Naïve Bayes binary classification → Accuracy validation.