Back office ops · Production

Building AI Agents: Lessons Learned over the past Year

The problem

Building production AI agents proved far harder than early prototypes suggested; agents initially struggled to generalize to all real-world scenarios across disparate data sources and businesses.

First attempt

Agents on weaker models hallucinated table names instead of searching for real ones; agents silently ignored columns listed in tool-call responses due to markdown formatting; fine-tuned models degraded reasoning by causing agents to short-cut their directions.

Workflow diagram · grounded in source
1
User submits objective
trigger
“starting a conversation with some objective and an agent system prompt”
2
Model called for completion
ai_action
“calling a model for a completion, handling any tool calls that the model wants to make, doing this in a loop”
3
Tool calls executed on data
integration
“tool calls for RAG over a metadata layer describing the contents of the database, and tool calls for analyzing the data via SQL, python, and data visualization”
4
SQL errors fed back to agent
feedback_loop
“We make sure to return any SQL errors, along with all the context we can capture, back to the agent when its query fails… which enables the agent to resolve the issue and get the code working a vast majority of the time”
5
Agent delivers final output
output
“stopping when it is done with its work”
Reported outcome

After extensive iteration, the team achieved a baseline of stability and performance across disparate data sources and businesses, with Fortune 500 users relying on the agent daily for data analysis.

Reported metrics
text-to-SQL benchmark accuracy ceiling80%
ACI iterationshundreds of times
SQL error resolution ratea vast majority of the time
New model deployment speedwithin 15 minutes
Reported stack
gpt-4oClaude OpusClaudeCommand R+gpt-4-turbogpt-3.5-turbogpt-4–32kRAGLangChainLlamaIndexSnowflakeBigQuery
Source
https://medium.com/@cpdough/building-ai-agents-lessons-learned-over-the-past-year-41dc4725d8e5
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After extensive iteration, the team achieved a baseline of stability and performance across disparate data sources and businesses, with Fortune 500 users relying on the agent daily for data analysis.

What tools did this team use?

gpt-4o, Claude Opus, Claude, Command R+, gpt-4-turbo, gpt-3.5-turbo, gpt-4–32k, RAG, LangChain, LlamaIndex.

What results were reported?

text-to-SQL benchmark accuracy ceiling: 80%; ACI iterations: hundreds of times; SQL error resolution rate: a vast majority of the time; New model deployment speed: within 15 minutes (source-reported, not independently verified).

What failed first in this deployment?

Agents on weaker models hallucinated table names instead of searching for real ones; agents silently ignored columns listed in tool-call responses due to markdown formatting; fine-tuned models degraded reasoning by ca…

How is this back office ops AI workflow structured?

User submits objective → Model called for completion → Tool calls executed on data → SQL errors fed back to agent → Agent delivers final output.

Related back office ops cases