Back office ops · Production

T-RAG: on-premise RAG with finetuned LLM and tree-based entity context for governance document QA

The problem

A large non-profit organization needed secure, on-premise question answering over its private governance documents but could not use API-based LLMs due to data leakage risks, faced limited computational resources, and required reliably correct responses to organizational queries.

Workflow diagram · grounded in source
1
PDF document ingested
integration
“The first step was to parse the original PDF document file into text format for further processing; this was done using the LangChain library”
2
Document chunked and embedded
ai_action
“The input document is split into discrete chunks. Using an encoder model, the split chunks are converted to embedding vectors which are then stored in a vector database”
3
QA dataset generated for finetuning
ai_action
“for each chunk we prompted the Llama-2 model to generate questions and answers for the provided chunk. Our dataset consists of 1,614 question and answer pairs that were randomly split into 90% training and 10% validation sets”
4
LLM finetuned with QLoRA
ai_action
“We finetuned the base LLM model on our Q&A instruction dataset with QLoRA, using the Hugging Face 'peft' library”
5
User query received
trigger
“The Query processing happens in response to incoming user queries”
6
Org entity detection in query
ai_action
“A parser module searches the user query for keywords matching the names of entities in the organization. We used the spaCy library for detecting and extracting such named entities from the user's query”
7
Tree context retrieval
routing
“If one or more matches are found, information about each matched entity is extracted from the tree and converted into a textual statement providing information about the entity and its location within the organization's hierarchy”
8
Augmented response generated
output
“This information is then combined with the document chunks retrieved from the vector database to form the context. This allows the model to access information about entities and their location within the organization's hierarchy when use…”
Reported outcome

T-RAG, combining RAG with a finetuned open-source LLM and a tree structure for organizational entity hierarchies, performs better than a simple RAG or finetuning implementation alone.

Reported metrics
T-RAG vs RAG or finetuning-only performanceperforms better than a simple RAG or finetuning implementation
Finetuning instruction dataset size1,614 question and answer pairs
trainable parameter reduction with QLoRAabout 200 times reduction compared to the full set of parameters in the model
Reported stack
LangChainLlama-2QLoRAvector database
Source
https://arxiv.org/html/2402.07483v2
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

T-RAG, combining RAG with a finetuned open-source LLM and a tree structure for organizational entity hierarchies, performs better than a simple RAG or finetuning implementation alone.

What tools did this team use?

LangChain, Llama-2, QLoRA, vector database.

What results were reported?

T-RAG vs RAG or finetuning-only performance: performs better than a simple RAG or finetuning implementation; Finetuning instruction dataset size: 1,614 question and answer pairs; trainable parameter reduction with QLoRA: about 200 times reduction compared to the full set of parameters in the model (source-reported, not independently verified).

How is this back office ops AI workflow structured?

PDF document ingested → Document chunked and embedded → QA dataset generated for finetuning → LLM finetuned with QLoRA → User query received → Org entity detection in query → Tree context retrieval → Augmented response generated.

Related back office ops cases