Customer support · Production
Assembled empirically compares LLM evaluation metrics for Assembled Assist, its AI customer support agent tool
The problem
As Assembled Assist grew, the team needed scalable automated methods to evaluate response quality, prevent regressions when updating the AI system, and measure improvements across hundreds of evaluation cases without relying solely on manual scoring.
Workflow diagram · grounded in source
1
Conversation context arrives
trigger
“Given some context of a conversation, Cal pulls in relevant information for agents”
2
Retrieve relevant information
ai_action
“Cal pulls in relevant information for agents and drafts a reply they can use in their conversations with customers”
3
Draft agent reply
output
“drafts a reply they can use in their conversations with customers”
4
Manual interaction scoring
human_review
“we manually score each Cal interaction. We want to ensure that Cal's tone is aligned with the company's tone and that the answers are accurate”
5
Automated evaluation metrics
validation
“More sophisticated evaluations that leverage embeddings (LLM Eval, Ragas, BERTScore) perform better than n-gram-based metrics (Rouge, Bleu)”
6
Version regression comparison
validation
“We can compare the new, prototype Cal vs. current Cal for both questions”
Reported outcome
LLM-based and embedding evaluations outperformed n-gram-based metrics, with Assembled's custom LLM Eval dominating quantitative approaches.
Automating evaluations is projected to save thousands of hours.
Reported metrics
Hours saved by automating evaluationsthousands of hours
Meta-evaluation dataset size40
Reported stack
Assembled Assist
Source
https://www.assembled.com/blog/ai-at-assembled-an-empirical-comparison-of-llm-evaluation-metrics-in-a-customer-support-setting
Read source ↗Frequently asked questions
What did this team achieve with this AI workflow?
LLM-based and embedding evaluations outperformed n-gram-based metrics, with Assembled's custom LLM Eval dominating quantitative approaches.
What tools did this team use?
Assembled Assist.
What results were reported?
Hours saved by automating evaluations: thousands of hours; Meta-evaluation dataset size: 40 (source-reported, not independently verified).
How is this customer support AI workflow structured?
Conversation context arrives → Retrieve relevant information → Draft agent reply → Manual interaction scoring → Automated evaluation metrics → Version regression comparison.