Quality assurance · Production
DREAM: A blueprint for distributed RAG experimentation using Ray, LlamaIndex, Ragas, MLFlow and MinIO on Kubernetes
The problem
Evaluating which combination of LLMs, embedding models, and retrieval methods works best for a RAG use case is time-consuming when combinations must be explored sequentially.
Workflow diagram · grounded in source
1
Load and store unstructured data
integration
“Download the PDFs from github to local - Use boto3 to push the PDFs to S3 (MinIO)”
2
Distributed golden dataset generation
ai_action
“With our Jupyter notebook acting as the Ray driver, we use the ray client to submit the Ray job for creating the golden dataset in a distributed manner. In each Ray task, upto 3 PDFs are loaded from S3 and then the ragas framework's Test…”
3
Ray Tune parameter search
ai_action
“our search space spans over 3x RAG methods, 2x LLMs and 2x embedding models”
4
RAG evaluation with ragas metrics
validation
“we use the ragas framework's faithfulness, answer_relevancy, context_precision, context_recall, answer_correctness and answer_similarity as metrics”
5
Experiment tracking with MLflow
output
“we leverage the amazing experiment tracking capability of MLflow to record experiment results, establish lineage with the golden dataset and visualise experiment results”
Reported outcome
DREAM demonstrates how distributed RAG experimentation, evaluation, and tracking can be conducted using open-source technologies, enabling comparison and selection of the best-performing RAG parameter combination.
Reported stack
Ray TunePostgreSQLboto3gpt-3.5-turbogpt-4text-embedding-3-smalltext-embedding-3-largeOpenAI
Frequently asked questions
What did this team achieve with this AI workflow?
DREAM demonstrates how distributed RAG experimentation, evaluation, and tracking can be conducted using open-source technologies, enabling comparison and selection of the best-performing RAG parameter combination.
What tools did this team use?
Ray Tune, PostgreSQL, boto3, gpt-3.5-turbo, gpt-4, text-embedding-3-small, text-embedding-3-large, OpenAI.
How is this quality assurance AI workflow structured?
Load and store unstructured data → Distributed golden dataset generation → Ray Tune parameter search → RAG evaluation with ragas metrics → Experiment tracking with MLflow.