Customer support · Production

What's So Challenging About Building Chatbots? Drawing lessons from the trenches.

The problem

Building enterprise chatbots proved far more complex than initial estimates: knowledge is scattered across disparate systems, RAG retrievers fail to surface the most relevant chunks, LLMs provide no native conversation flow or persistent memory, and regulatory compliance adds further constraints in sectors like finance and healthcare.

First attempt

A bank's chatbot proof-of-concept planned for a Q3 launch was still struggling a year later. OpenAI.com's own chatbot proved problematic, often requiring human intervention for straightforward issues, and eventually shifted to predefined response options instead of free-flowing conversations.

Workflow diagram · grounded in source
1
Domain knowledge assembly
integration
“From Apache Spark to PDF manuals to previous chat logs to undocumented ideas is in the brains of the agent, there is a lot of data needed to solve the problem”
2
RAG retrieval
ai_action
“You take your data, send it to a AI model that will generate a big list of numbers and then you can use them to do similarity search. Rewording the famous episode from the Seinfeld, the Vector DBs know how to take in the data, but just n…”
3
Conversation flow routing
routing
“It can be a combination of rules that we humans learned over time combined with machine learning techniques that would allow you to predict what should be next thing to ask. These are often represented as decision trees”
4
Conversation state storage
integration
“We need to store some of this data as tags in a key value store. Some of the parts we need to store in a knowledge graphs where the relationships between different parts of the conversation can be maintained. Some other parts have to jus…”
Reported outcome

(not stated)

Reported stack
GPT-4PineconeRAGGroqApache Spark
Source
https://medium.com/@balajivis/whats-so-challenging-about-building-chatbots-drawing-lessons-from-the-trenches-1ca7343c6e3d
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

(not stated)

What tools did this team use?

GPT-4, Pinecone, RAG, Groq, Apache Spark.

What failed first in this deployment?

A bank's chatbot proof-of-concept planned for a Q3 launch was still struggling a year later.

How is this customer support AI workflow structured?

Domain knowledge assembly → RAG retrieval → Conversation flow routing → Conversation state storage.

Related customer support cases