Back office ops · Production

Building a RAG system for internal engineering knowledge search from 1 TB of project documents

The problem

An engineering company needed an internal natural language chat tool to search across nearly a decade of project history totaling 1 TB of mixed technical documents—including OrcaFlex simulation files central to the offshore industry—without relying on external APIs for confidentiality reasons.

First attempt

LlamaIndex overflowed the laptop's RAM when processing large non-text files; a custom checkpoint system suffered data corruption and was too slow; the laptop's integrated GPU required 4-5 hours per 500 MB; and the production VM had only 100 GB of disk, far short of the full document corpus.

Workflow diagram · grounded in source
1
Document filtering by extension and pattern
validation
“I added a filtering system to the pipeline that excluded files by extension and by name patterns (simulation files, numerical results, etc.)”
2
Batch embedding and vector storage
ai_action
“a batch pipeline that processed 150 files at a time, generated their embeddings, and stored them directly in ChromaDB. This allowed indexing the 451GB of documents across multiple sessions, with checkpoints, without losing progress on in…”
3
User submits natural language question
trigger
“It also has a native widget for Q&A, in the style of any current chat for interacting with an AI”
4
RAG query and LLM response generation
ai_action
“With Flask I built a simple API to access LlamaIndex, which in turn queried ChromaDB and Ollama”
5
Response with source document references
output
“the system must show the information sources, that is, the documents used to generate the answer”
6
Document download via SAS token from Azure
integration
“For each document cited in a response, the system generates a download link with a SAS token that allows the user to download it directly from the cloud.”
Reported outcome

The RAG system reached production with 738,470 vectors and a 54 GB index in ChromaDB, achieved a 54% reduction in files to index through filtering, and is described as fast, reliable, and useful for colleagues.
The GPU indexing phase cost 184 euros on Hetzner.

Reported metrics
Files to index reduction54%
vectors in ChromaDB index738,470
ChromaDB index size54GB
GPU rental cost on Hetzner184 euros
Show all 10 reported metrics
files to index reduction54%
vectors in ChromaDB index738,470
ChromaDB index size54GB
GPU rental cost on Hetzner184 euros
CPU indexing speed before GPU4-5 hours per 500 MB
total GPU indexing durationbetween 2 and 3 weeks
document corpus size451GB
production VM disk space100 GB
original project data volume1 TB
system qualityfast, reliable, and above all useful
Reported stack
Ollamanomic-embed-textLlamaIndexChromaDBSQLiteFlaskStreamlitPythonAzure Blob StorageDocker ComposeNVIDIA Container ToolkitGunicornllama3.2:3b
Source
https://en.andros.dev/blog/aa31d744/from-zero-to-a-rag-system-successes-and-failures/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The RAG system reached production with 738,470 vectors and a 54 GB index in ChromaDB, achieved a 54% reduction in files to index through filtering, and is described as fast, reliable, and useful for colleagues.

What tools did this team use?

Ollama, nomic-embed-text, LlamaIndex, ChromaDB, SQLite, Flask, Streamlit, Python, Azure Blob Storage, Docker Compose.

What results were reported?

Files to index reduction: 54%; vectors in ChromaDB index: 738,470; ChromaDB index size: 54GB; GPU rental cost on Hetzner: 184 euros (source-reported, not independently verified).

What failed first in this deployment?

LlamaIndex overflowed the laptop's RAM when processing large non-text files; a custom checkpoint system suffered data corruption and was too slow; the laptop's integrated GPU required 4-5 hours per 500 MB; and the pro…

How is this back office ops AI workflow structured?

Document filtering by extension and pattern → Batch embedding and vector storage → User submits natural language question → RAG query and LLM response generation → Response with source document references → Document download via SAS token from Azure.

Related back office ops cases