Customer support · Production

RAG with Knowledge Graphs reduces median customer service issue resolution time by 28.6% at LinkedIn

The problem

Conventional RAG for customer service treated historical issue tracking tickets as plain text, discarding intra-issue structure and inter-issue relations. Text segmentation further disconnected related content such as a ticket's problem description and its solution, degrading both retrieval accuracy and answer quality.

First attempt

The baseline embedding-based retrieval approach lost structural ticket relationships by compressing documents into plain-text chunks, and text segmentation split tickets such that problem descriptions and their solutions appeared in separate chunks, causing incomplete answers.

Workflow diagram · grounded in source
1
KG construction from historical tickets
ai_action
“Our method constructs a KG from historical issues for use in retrieval, retaining the intra-issue structure and inter-issue relations”
2
Consumer query submitted
trigger
“our method parses consumer queries to identify named entities and intents”
3
Query entity and intent identification
ai_action
“we extract the named entities of type and the query intent set from each user query . The method involves parsing each query into a key-value pair, where each key , mentioned within the query, corresponds to an element in the graph template”
4
Embedding-based sub-graph retrieval
ai_action
“Our method extracts pertinent sub-graphs from the knowledge graph, aligned with user-provided specifics such as "issue description" and "issue summary", as well as user intentions like "fix solution"”
5
Graph query translation and retrieval
ai_action
“the system first rephrases the original user query to include the retrieved ticket ID; the modified query is then translated into a graph database language, such as Cypher for Neo4j for question answering”
6
Answer generation by LLM
output
“The LLM serves as a decoder to formulate responses to user inquiries given the retrieved information”
7
Fallback to text-based retrieval
routing
“if query execution encounters issues, a fallback mechanism reverts to a baseline text-based retrieval method”
Reported outcome

The KG-augmented RAG system outperformed the baseline by 77.6% in MRR and by 0.32 in BLEU.
Deployed at LinkedIn's customer service team, the system reduced median per-issue resolution time by 28.6%, with mean resolution time dropping from 40 hours to 15 hours.

Reported metrics
MRR improvement over baseline77.6%
BLEU score improvement over baseline0.32
Median per-issue resolution time reduction28.6%
Mean resolution time — tool not used40 Hours
Show all 9 reported metrics
MRR improvement over baseline77.6%
BLEU score improvement over baseline0.32
median per-issue resolution time reduction28.6%
mean resolution time — tool not used40 Hours
mean resolution time — tool used15 hours
P50 (median) resolution time — tool not used7 Hours
P50 (median) resolution time — tool used5 hours
P90 resolution time — tool not used87 Hours
P90 resolution time — tool used47 hours
Reported stack
LLMGPT-4BERTE5QdrantNeo4jJira
Source
https://arxiv.org/html/2404.17723
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The KG-augmented RAG system outperformed the baseline by 77.6% in MRR and by 0.32 in BLEU.

What tools did this team use?

LLM, GPT-4, BERT, E5, Qdrant, Neo4j, Jira.

What results were reported?

MRR improvement over baseline: 77.6%; BLEU score improvement over baseline: 0.32; Median per-issue resolution time reduction: 28.6%; Mean resolution time — tool not used: 40 Hours (source-reported, not independently verified).

What failed first in this deployment?

The baseline embedding-based retrieval approach lost structural ticket relationships by compressing documents into plain-text chunks, and text segmentation split tickets such that problem descriptions and their soluti…

How is this customer support AI workflow structured?

KG construction from historical tickets → Consumer query submitted → Query entity and intent identification → Embedding-based sub-graph retrieval → Graph query translation and retrieval → Answer generation by LLM → Fallback to text-based retrieval.

Related customer support cases