Quality assurance · Production

Stack Overflow builds Question Assistant using logistic regression and Gemini to raise question success rate by 12%

The problem

Staging Ground reviewers were repeating the same feedback comments over and over on new askers' question drafts, slowing the review process, and LLMs alone proved unreliable for rating question quality.

First attempt

An LLM-only approach produced repetitive, category-agnostic feedback that did not change when question drafts were updated. A survey-based attempt to build a ground truth dataset yielded a low Krippendorff's alpha score, making the labeled data unusable for training reliable ML models.

Workflow diagram · grounded in source
1
Question draft submitted
trigger
“we ran this experiment in two stages: first on Staging Ground only, then on stackoverflow.com for all question askers with Ask Wizard”
2
Logistic regression flags indicators
ai_action
“we created individual logistic regression models. These produce a binary response based on the question title and body. Essentially: Does this question need a specific comment template applied to it or not?”
3
Gemini synthesizes contextual feedback
ai_action
“Once an indicator flags a question, it sends a preloaded response text with the question to Gemini, along with some system prompts. Gemini then synthesizes these to produce feedback that addresses the indicator, but is specific to the qu…”
4
Actionable feedback delivered to asker
output
“provide more specific, contextual feedback that is actionable for the asker in improving their question”
5
Results logged for model improvement
feedback_loop
“we collected events through Azure Event Hub and logged predictions and results to Datadog to understand whether or not the generated feedback was helpful for the user, and to improve future iterations of the indicator models”
Reported outcome

Question Assistant achieved a steady success rate improvement of +12% across two A/B test experiments, validating positive impact on question quality, and was released to all Stack Overflow askers on March 6, 2025.

Reported metrics
Question success rate+12%
Survey participants completing the survey152
Survey recipients1,000
A/B test group split50/50
Reported stack
GeminiAzure DatabricksAzure KubernetesDatabricks Unity CatalogAzure Event HubDatadogTF IDFAsk WizardStaging GroundGoogle
Source
https://stackoverflow.blog/2025/03/12/a-look-under-the-hood-how-and-why-we-built-question-assistant/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Question Assistant achieved a steady success rate improvement of +12% across two A/B test experiments, validating positive impact on question quality, and was released to all Stack Overflow askers on March 6, 2025.

What tools did this team use?

Gemini, Azure Databricks, Azure Kubernetes, Databricks Unity Catalog, Azure Event Hub, Datadog, TF IDF, Ask Wizard, Staging Ground, Google.

What results were reported?

Question success rate: +12%; Survey participants completing the survey: 152; Survey recipients: 1,000; A/B test group split: 50/50 (source-reported, not independently verified).

What failed first in this deployment?

An LLM-only approach produced repetitive, category-agnostic feedback that did not change when question drafts were updated.

How is this quality assurance AI workflow structured?

Question draft submitted → Logistic regression flags indicators → Gemini synthesizes contextual feedback → Actionable feedback delivered to asker → Results logged for model improvement.

Related quality assurance cases