Quality assurance · Production

GraphRAG increases faithfulness but not other RAGAS metrics — ROI of knowledge graphs may not justify the hype

The problem

The analysis sought to determine whether GraphRAG (knowledge graph creation and retrieval via Neo4j using Cypher) provides meaningfully better performance over vector-based RAG (FAISS) across standard RAGAS metrics, given its additional setup and maintenance complexity.

Workflow diagram · grounded in source
1
Knowledge graph creation
ai_action
“This approach uses GPT-3.5-Turbo to extract entities and relationships from our text, creating a dynamic knowledge graph based on the content of our document”
2
Retriever configuration
integration
“We'll set up two types of retrievers: one using FAISS for vector-based retrieval, and another using Neo4j for graph-based retrieval. The FAISS retriever uses vector similarity to find relevant information, while the Neo4j retrievers leve…”
3
RAG chain evaluation
validation
“To evaluate our RAG systems, we'll create a ground truth dataset and use the RAGAS framework”
4
Results comparison output
output
“This analysis revealed a surprising similarity in performance between GraphRAG and vector-based RAG across most metrics, with one difference: - Faithfulness: - Neo4j GraphRAG significantly outperformed FAISS (0.54 vs 0.18)”
Reported outcome

GraphRAG significantly outperformed FAISS on faithfulness (0.54 vs 0.18) but showed surprising similarity across all other RAGAS metrics, suggesting the added complexity of a graph database may not always be justified.

Reported metrics
GraphRAG (Neo4j) faithfulness score0.54
FAISS faithfulness score0.18
Reported stack
Neo4jFAISSGPT-3.5-TurboRAGASCypher
Source
https://mlops.community/blog/graphrag-analysis-part-2-graph-creation-and-retrieval-vs-vector-database-retrieval
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

GraphRAG significantly outperformed FAISS on faithfulness (0.54 vs 0.18) but showed surprising similarity across all other RAGAS metrics, suggesting the added complexity of a graph database may not always be justified.

What tools did this team use?

Neo4j, FAISS, GPT-3.5-Turbo, RAGAS, Cypher.

What results were reported?

GraphRAG (Neo4j) faithfulness score: 0.54; FAISS faithfulness score: 0.18 (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Knowledge graph creation → Retriever configuration → RAG chain evaluation → Results comparison output.

Related quality assurance cases