Back office ops · Production

Dataherald cuts LLM costs by 83% by detecting uncapped token growth with LangSmith

The problem

Dataherald's NL-to-SQL engine required GPT-4–32K for accuracy, making token costs a central concern. Their manual tracking approach via MongoDB could not identify which agent tools were driving costs, and a bug in the few-shot retriever caused token usage to grow practically uncapped over time.

First attempt

Their prior monitoring stack — TikToken library and LangChain callback handlers written to MongoDB — required manual queries in MongoDB Compass to aggregate costs, gave no per-tool visibility, and caused them to miss a growing bug in the few-shot retriever.

Workflow diagram · grounded in source
1
Plain English query via API
trigger
“It allows you to set up APIs to answer plain English questions from your relational database”
2
RAG agent reasons with LLM
ai_action
“the Dataherald NL-to-SQL engine is a RAG agent implemented using LangChain. While our agent uses the LLM for reasoning, we have implemented a range of tools for it to use, including few shot samples stored in a vector DB, complex automat…”
3
LangSmith captures execution metrics
integration
“monitor your agent's execution process, providing essential metrics such as agent token usage and latency broken down by each tool”
4
Bug detected: uncapped token growth
validation
“Within hours after setting up Langsmith we realized our few shot sample retriever's token usage was growing uncontrollably. On average, we were using about 150,000 tokens per query”
5
Hotfix reduces token usage by 83%
output
“We pushed a hotfix immediately and in the same day identified a few more fixes to drive down costs even further. Thanks to these adjustments, our token usage dropped to approximately 25,500 tokens per query, slashing our costs by 83%”
6
Run links shared for ongoing debugging
feedback_loop
“rarely a day goes by without links to Langsmith execution runs being Slacked around between team members and instead of trying to recreate issues in a dev or staging environment we can simply add a link to the Jira ticket to point someon…”
Reported outcome

Within hours of setting up LangSmith, Dataherald identified a bug causing the few-shot retriever to consume about 150,000 tokens per query.
After a hotfix and additional fixes, token usage dropped to approximately 25,500 tokens per query, slashing costs by 83%. The team now shares LangSmith execution run links via Slack and Jira for rapid production debugging.

Reported metrics
LLM cost reduction83%
Tokens per query before fix150,000
Tokens per query after fix25,500
Bug identification and triage timefraction of the time
Reported stack
LangSmithLangChainRAGGPT-4–32Kvector DBTikTokenMongoDB CompassSlackJira
Source
https://medium.com/dataherald/cutting-llm-costs-by-83-with-langsmith-e44bb63af2a8
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Within hours of setting up LangSmith, Dataherald identified a bug causing the few-shot retriever to consume about 150,000 tokens per query.

What tools did this team use?

LangSmith, LangChain, RAG, GPT-4–32K, vector DB, TikToken, MongoDB Compass, Slack, Jira.

What results were reported?

LLM cost reduction: 83%; Tokens per query before fix: 150,000; Tokens per query after fix: 25,500; Bug identification and triage time: fraction of the time (source-reported, not independently verified).

What failed first in this deployment?

Their prior monitoring stack — TikToken library and LangChain callback handlers written to MongoDB — required manual queries in MongoDB Compass to aggregate costs, gave no per-tool visibility, and caused them to miss…

How is this back office ops AI workflow structured?

Plain English query via API → RAG agent reasons with LLM → LangSmith captures execution metrics → Bug detected: uncapped token growth → Hotfix reduces token usage by 83% → Run links shared for ongoing debugging.

Related back office ops cases