Customer support · Production

GitHub builds an internal AI analytics tool to transform support ticket feedback into product insights

The problem

GitHub receives large volumes of customer feedback through its support portal daily, and manually sifting through the text data was an overwhelming, error-prone, and time-consuming challenge that left many customer insights untapped.

Workflow diagram · grounded in source
1
Customer feedback received
trigger
“We receive countless pieces of feedback through our support portal every day.”
2
BERT document embeddings
ai_action
“BERTopic is an open source topic modeling framework that leverages Bidirectional Encoder Representations from Transformers (BERT) embeddings to create dynamic and interpretable topics.”
3
HDBSCAN document clustering
ai_action
“BERTopic combines BERT's ability to generate high-quality document embeddings with a clustering algorithm, typically Hierarchical Density-Based Spatial Clustering of Applications with Noise, (HDBSCAN), to group similar documents together.”
4
Representative word extraction
ai_action
“The topics are then derived by extracting and aggregating the most representative words from each cluster.”
5
GPT-4 cluster summarization
ai_action
“To ensure we could display the customer feedback insights in a more comprehensible form, we decided to summarize them using GPT-4, a powerful and popular large language model (LLM).”
6
Iterative human feedback
feedback_loop
“Continuously improving the model's performance through human feedback and A/B testing.”
7
Internal tool displays insights
output
“create a unique internal AI analytics tool that presents the most relevant and actionable trends, complete with business context specifically tailored to GitHub's product areas.”
8
Product team decision-making
integration
“By taking advantage of the clear, summarized insights our tool provides, our internal teams can make more informed decisions that are more aligned with the needs and desires of our developer community.”
Reported outcome

Automated AI-driven trend identification significantly enhanced GitHub's ability to scale data analysis, saves time, increases precision in understanding developer feedback, helps identify recurring pain points quicker, and enables more informed product prioritization decisions.

Reported metrics
Data scientist time on data tasks (external benchmark cited)80%
Data analysis scalesignificantly enhanced our ability to scale our data analysis efforts
Time to identify recurring problemsidentify recurring problems quicker
User dependency on direct supportreducing dependency on direct support
Reported stack
BERTopicBERTHDBSCANGPT-4Azure Data Explorer
Source
https://github.blog/ai-and-ml/machine-learning/how-github-harnesses-ai-to-transform-customer-feedback-into-action/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Automated AI-driven trend identification significantly enhanced GitHub's ability to scale data analysis, saves time, increases precision in understanding developer feedback, helps identify recurring pain points quicke…

What tools did this team use?

BERTopic, BERT, HDBSCAN, GPT-4, Azure Data Explorer.

What results were reported?

Data scientist time on data tasks (external benchmark cited): 80%; Data analysis scale: significantly enhanced our ability to scale our data analysis efforts; Time to identify recurring problems: identify recurring problems quicker; User dependency on direct support: reducing dependency on direct support (source-reported, not independently verified).

How is this customer support AI workflow structured?

Customer feedback received → BERT document embeddings → HDBSCAN document clustering → Representative word extraction → GPT-4 cluster summarization → Iterative human feedback → Internal tool displays insights → Product team decision-making.

Related customer support cases