Customer support · Production

Podium reduces engineering intervention by 90% and improves AI Employee accuracy using LangSmith

The problem

Podium's AI Employee processed 20-30 LLM calls per interaction, making it hard to understand agent behavior or debug issues without engineering involvement. The TPS support team lacked visibility into LLM inputs and outputs needed to resolve customer-reported problems independently.

First attempt

The AI Employee struggled to recognize when a conversation had naturally ended, resulting in awkward repeated goodbyes. Resolving such issues required calling in engineers to review model inputs and outputs and rewrite code.

Workflow diagram · grounded in source
1
Customer inquiry to AI Employee
trigger
“Podium launched AI Employee, their agentic application (and flagship product) to engage local business customers, schedule appointments, and close sales”
2
Multi-turn LLM conversation
ai_action
“the Podium engineers made 20-30 LLM calls per interaction”
3
LangSmith trace logging
integration
“Podium engineers also enriched LangSmith traces with metadata on customer profiles, business types, and other parameters important to their business. They grouped traces using specific identifiers in LangSmith, making it easy to aggregat…”
4
TPS team issue diagnosis
human_review
“Giving the TPS team access to LangSmith provided clarity, allowing the team to quickly identify customer-reported issues and determine: "Is this issue caused by a bug in the application, incomplete context, misaligned instructions, or an…”
5
Dataset curation and model fine-tuning
feedback_loop
“engineering team then found it helpful to upgrade to a larger model, curating the outputs into a smaller model (using a technique called model distillation). Upgrading their model went smoothly since model inputs and outputs were automat…”
6
Pairwise evaluation
validation
“the Podium team then compared the results from their fine-tuned model against results from their original, larger model using pairwise evaluations”
Reported outcome

After fine-tuning with LangSmith-curated datasets, Podium's AI Employee F1 scores improved by 7.5% from 91.7% to 98.6%, exceeding their quality threshold.
Engineering intervention for support issues was reduced by 90%, and customer satisfaction scores improved.

Reported metrics
Lead conversion rate uplift (5-min response vs 1-hour)46% higher
F1 score improvement after fine-tuning7.5%
F1 score before fine-tuning91.7%
F1 score after fine-tuning98.6%
Show all 7 reported metrics
lead conversion rate uplift (5-min response vs 1-hour)46% higher
F1 score improvement after fine-tuning7.5%
F1 score before fine-tuning91.7%
F1 score after fine-tuning98.6%
engineering intervention reduction90%
CSAT scoresImproved customer satisfaction (CSAT) scores
LLM calls per interaction20-30 LLM calls per interaction
Reported stack
LangChainLangSmithLangGraph
Source
https://blog.langchain.dev/customers-podium/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After fine-tuning with LangSmith-curated datasets, Podium's AI Employee F1 scores improved by 7.5% from 91.7% to 98.6%, exceeding their quality threshold.

What tools did this team use?

LangChain, LangSmith, LangGraph.

What results were reported?

Lead conversion rate uplift (5-min response vs 1-hour): 46% higher; F1 score improvement after fine-tuning: 7.5%; F1 score before fine-tuning: 91.7%; F1 score after fine-tuning: 98.6% (source-reported, not independently verified).

What failed first in this deployment?

The AI Employee struggled to recognize when a conversation had naturally ended, resulting in awkward repeated goodbyes.

How is this customer support AI workflow structured?

Customer inquiry to AI Employee → Multi-turn LLM conversation → LangSmith trace logging → TPS team issue diagnosis → Dataset curation and model fine-tuning → Pairwise evaluation.

Related customer support cases