customer_support · saas · workflow
Mintlify rebuilds feedback pipeline and uses LLM classification to identify AI assistant weaknesses
The Mintlify AI assistant was underperforming in ways not yet understood, and there was no way to map negative feedback events back to their original conversation threads to diagnose failures systematically.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Thumbs-down events trigger analysis
Thumbs down events on assistant messages serve as the starting signal for the improvement analysis.
Tools used
ClickHouseLLMSonnet 4.5PSQL
Outcome
The analysis identified search quality as the assistant's biggest weakness, the assistant insights tab was expanded to surface LLM-categorized conversations to doc owners, and several UI improvements were shipped.
What failed first
Feedback events were stored in ClickHouse but could not be linked to their original conversation threads, and the PSQL setup made direct querying impossible, blocking any systematic analysis of negative interactions.
Results
Volumeabout 100
Grounding & classification
Source type: technical build writeup
22 fields verified against source quotes, 1 dropped as unverifiable.
document classificationknowledge searchsupport agentchat transcriptknowledge basefailure mode describedproduction runtime claimedtools describedworkflow describedsoftwareaccuracy improvementtechnical build writeupcustomer supportquality assuranceextract classify route