Customer support · Production

Mintlify rebuilds feedback pipeline and uses LLM classification to identify AI assistant weaknesses

The problem

The Mintlify AI assistant was underperforming in ways not yet understood, and there was no way to map negative feedback events back to their original conversation threads to diagnose failures systematically.

First attempt

Feedback events were stored in ClickHouse but could not be linked to their original conversation threads, and the PSQL setup made direct querying impossible, blocking any systematic analysis of negative interactions.

Workflow diagram · grounded in source
1
Thumbs-down events trigger analysis
trigger
“looking into trends on thumbs down events on assistant messages to understand where the experience was breaking down”
2
Feedback pipeline rebuilt to ClickHouse
integration
“we updated the server so that when it receives a feedback event, it pushes the full conversation thread to ClickHouse. Previously, this was only happening on the client side. We then ran a backfill script to copy all messages with feedba…”
3
Manual review and category creation
human_review
“We read through about 100 negative conversation threads and created eight categories for the different types of feedback”
4
LLM classifies conversation sample
ai_action
“we took a random sample of 1,000 conversations and used an LLM to classify each one into those eight buckets”
5
Search weakness identified
output
“This highlights search across the docs as one of the assistant's biggest weaknesses. Anecdotal feedback and our usage patterns point to the same conclusion”
6
Insights tab and UI improvements shipped
output
“We expanded the assistant insights tab in the dashboard by surfacing conversations that LLMs automatically categorized. This gives doc owners a way to sift through conversations and get a clearer view of what customers are confused about…”
Reported outcome

The analysis identified search quality as the assistant's biggest weakness, the assistant insights tab was expanded to surface LLM-categorized conversations to doc owners, and several UI improvements were shipped.

Reported metrics
Negative conversation threads manually reviewedabout 100
Feedback categories createdeight
conversations LLM-classified1,000
Overall assistant response quality assessmentimpressed with the overall quality of the assistant's responses
Reported stack
ClickHouseLLMSonnet 4.5PSQL
Source
https://www.mintlify.com/blog/assistant-improvements
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The analysis identified search quality as the assistant's biggest weakness, the assistant insights tab was expanded to surface LLM-categorized conversations to doc owners, and several UI improvements were shipped.

What tools did this team use?

ClickHouse, LLM, Sonnet 4.5, PSQL.

What results were reported?

Negative conversation threads manually reviewed: about 100; Feedback categories created: eight; conversations LLM-classified: 1,000; Overall assistant response quality assessment: impressed with the overall quality of the assistant's responses (source-reported, not independently verified).

What failed first in this deployment?

Feedback events were stored in ClickHouse but could not be linked to their original conversation threads, and the PSQL setup made direct querying impossible, blocking any systematic analysis of negative interactions.

How is this customer support AI workflow structured?

Thumbs-down events trigger analysis → Feedback pipeline rebuilt to ClickHouse → Manual review and category creation → LLM classifies conversation sample → Search weakness identified → Insights tab and UI improvements shipped.

Related customer support cases