Customer support · Production

10 Lessons from Developing an AI Chatbot Using Retrieval-Augmented Generation

The problem

Fiddler users needed a way to easily find answers from the company's documentation, while the development team faced challenges around LLM context window limits, diverse natural language query patterns, and chatbot hallucinations.

First attempt

During development the chatbot hallucinated by misinterpreting the acronym 'LLM' as 'local linear model' instead of 'large language model', highlighting a gap in the knowledge base; initial static block response formatting also felt disjointed to users.

Workflow diagram · grounded in source
1
User submits documentation query
trigger
“to help Fiddler users easily find answers from our documentation”
2
LangChain query preprocessing
ai_action
“Another example of LangChain's benefit is its capability to handle the preprocessing of user questions. This capability is critical in ensuring that queries are interpreted correctly by the chatbot”
3
RAG multi-retrieval
ai_action
“using multiple retrievals is crucial. This approach uses several searches instead of just one search to find the most relevant and complete information for accurate and helpful responses”
4
GPT-3.5 response generation
ai_action
“We used OpenAI's GPT-3.5 and augmented it with RAG to build the Fiddler Chatbot”
5
Streaming response delivery
output
“Streaming responses from the LLM in real-time, as if the chatbot is typing them out, created a more dynamic and engaging experience”
6
User feedback collection
feedback_loop
“offer a simple and quick feedback option, like a thumbs up/thumbs down (👍👎) button, along with every chatbot response”
7
LLM Observability monitoring
validation
“With the help of Fiddler LLM Observability solutions, we have continuously evaluated and monitored the chatbot's responses, user prompts”
8
Knowledge base enrichment
feedback_loop
“By enriching our knowledge base with detailed documentation about these concepts, the chatbot's response accuracy improved significantly”
Reported outcome

Fiddler deployed a RAG-based documentation chatbot using GPT-3.5 and LangChain, continuously monitored with Fiddler LLM Observability.
Hallucinations were mitigated through iterative knowledge base enrichment, and switching to streaming responses significantly enhanced user trust and conversational experience.

Reported metrics
Chatbot response accuracyimproved significantly
User trust in chatbotincreased their trust in the chatbot
Developer time and resourcessave considerable time and resources
Reported stack
LangChainGPT-3.5Fiddler LLM ObservabilityRAG
Source
https://www.fiddler.ai/resources/10-lessons-from-developing-an-ai-chatbot-using-retrieval-augmented-generation
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Fiddler deployed a RAG-based documentation chatbot using GPT-3.5 and LangChain, continuously monitored with Fiddler LLM Observability.

What tools did this team use?

LangChain, GPT-3.5, Fiddler LLM Observability, RAG.

What results were reported?

Chatbot response accuracy: improved significantly; User trust in chatbot: increased their trust in the chatbot; Developer time and resources: save considerable time and resources (source-reported, not independently verified).

What failed first in this deployment?

During development the chatbot hallucinated by misinterpreting the acronym 'LLM' as 'local linear model' instead of 'large language model', highlighting a gap in the knowledge base; initial static block response forma…

How is this customer support AI workflow structured?

User submits documentation query → LangChain query preprocessing → RAG multi-retrieval → GPT-3.5 response generation → Streaming response delivery → User feedback collection → LLM Observability monitoring → Knowledge base enrichment.

Related customer support cases