Back office ops · Production
How to build a scalable RAG-based enterprise knowledge assistant
The problem
Productizing and scaling LLM-based knowledge assistants for tens of thousands of enterprise users involves challenges across server capacity, algorithm tuning, robustness, reliability, privacy, and security.
Workflow diagram · grounded in source
1
User query received via API
trigger
“The Application Layer of the KA activates upon receiving a user query through an API”
2
Input guardrails applied
validation
“input guardrails encompass a selection of policies, business rules, and validations, and are designed to ensure that incoming requests meet predefined criteria and may proceed. The primary objective of these guardrails is to prevent user…”
3
Query improvement
ai_action
“The implementation should include a comprehensive glossary of company-specific terms and abbreviations. This glossary "translates" and modifies user queries for optimal retrieval. For instance, it could remove trailing punctuation and ex…”
4
Intent recognition
ai_action
“This method is highly effective for setups with fewer than ten intents, achieving over 90% accuracy”
5
Data source routing
routing
“Data Source Routing module determines where the KA receives its knowledge based on the user's intent. With the user's intent, the KA picks between three primary data sources”
6
Semantic or keyword retrieval
ai_action
“the retriever will either embed the question through LLM Gateway and perform a semantic similarity search, use it for a keyword search, or pass it to an external API that handles the retrieval”
7
Context enrichment
ai_action
“context enrichment phase requires crafting effective prompts. These prompts must harmoniously blend instructions from the assistant, retrieved context, and the chat history”
8
LLM answer generation
ai_action
“the next step answers the user's question using an LLM. As a reminder, the LLM prompt needs to include data from the databases (see Data Source Routing and Retriever) and conversation history”
9
Output guardrails
validation
“Output Guardrails in place. These are a set of asynchronously executed checks on the model's response that include a content filter and a hallucination detector”
10
Response delivered to user
output
“if the chat completion passes the output guardrails, the final step formats the response, and sends it to the front end for the user to read”
Reported outcome
(not stated)
Reported metrics
Intent recognition accuracyover 90%
Reported stack
RAGKedroLangChaingpt-4React
Frequently asked questions
What did this team achieve with this AI workflow?
(not stated)
What tools did this team use?
RAG, Kedro, LangChain, gpt-4, React.
What results were reported?
Intent recognition accuracy: over 90% (source-reported, not independently verified).
How is this back office ops AI workflow structured?
User query received via API → Input guardrails applied → Query improvement → Intent recognition → Data source routing → Semantic or keyword retrieval → Context enrichment → LLM answer generation → Output guardrails → Response delivered to user.