quality_assurance · workflow
GraphRAG increases faithfulness but not other RAGAS metrics — ROI of knowledge graphs may not justify the hype
The analysis sought to determine whether GraphRAG (knowledge graph creation and retrieval via Neo4j using Cypher) provides meaningfully better performance over vector-based RAG (FAISS) across standard RAGAS metrics, given its additional setup and maintenance complexity.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Knowledge graph creation
GPT-3.5-Turbo extracts entities and relationships from the document text to create a dynamic knowledge graph.
Tools used
Neo4jFAISSGPT-3.5-TurboRAGASCypher
Outcome
GraphRAG significantly outperformed FAISS on faithfulness (0.54 vs 0.18) but showed surprising similarity across all other RAGAS metrics, suggesting the added complexity of a graph database may not always be justified.
Results
Volume0.54
Grounding & classification
Source type: technical build writeup
16 fields verified against source quotes.
data extractionknowledge searchragknowledge basemetric backedsource backedtools describedworkflow describedsoftwareaccuracy improvementtechnical build writeupquality assurancerag answering