Quality assurance · Production

How Cisco Built End-to-End LLM Observability for Its Splunk AI Assistant Using RAG

The problem

Running LLM-powered applications at scale brings unique challenges around accuracy, reliability, cost control, and user trust, with no unified visibility into the full lifecycle of a RAG system's answers across retrieval, generation, and output quality.

First attempt

Without explicit prompt guidance, the RAG system failed to prioritize the most relevant document for a user query, producing an incomplete or potentially misleading answer—a mild hallucination—that required observability tooling to detect and diagnose.

Workflow diagram · grounded in source

User query submitted

trigger

“track the full lifecycle of an AI answer—from the user query, through document retrieval and prompt construction, to the LLM's final response”

RAG document retrieval

ai_action

“Retrieval-Augmented Generation (RAG) system to provide instant, accurate answers to FAQs using curated public content”

Source reliability classification

validation

“Source documents are classified into reliability tiers (green/yellow/red) based on predefined quality criteria”

LLM response generation

ai_action

“through the retrieved documents, the LLM prompt, response latency, token usage (cost), and even which model version handled the request”

Structured observability log capture

integration

“Structured logs that include prompts, sources, and trace IDs enable precise dashboards, alerting, and root-cause analysis in Splunk”

Dashboard monitoring and alerting

output

“This dashboard correlates key metrics—response quality, model latency, document reliability, and cost—in one unified view”

Root cause analysis

feedback_loop

“If a response is marked low quality, this dashboard supports end-to-end investigation—from the user question, through the retrieved documents, the LLM prompt, response latency, token usage (cost), and even which model version handled the…”

Reported outcome

Cisco deployed a Splunk-based observability system for its RAG pipeline that achieves a 99.982% success rate and provides end-to-end traceability from user query through retrieval, generation, and output quality, enabling rapid root-cause analysis of AI failures.

Reported metrics

RAG pipeline success rate99.982%

Pod restartsZero restarts

MTTR for LLM failure casesreduces mean time to resolution (MTTR) for LLM failure cases

Hallucination riskReduce hallucinations

Reported stack

SplunkRAGCIRCUITSplunk Observability CloudSplunk SearchSPLBridgeIT RAG-as-a-Service

Source

https://www.splunk.com/en_us/blog/artificial-intelligence/how-we-built-end-to-end-llm-observability-with-splunk-and-rag.html

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Cisco deployed a Splunk-based observability system for its RAG pipeline that achieves a 99.982% success rate and provides end-to-end traceability from user query through retrieval, generation, and output quality, enab…

What tools did this team use?

Splunk, RAG, CIRCUIT, Splunk Observability Cloud, Splunk Search, SPL, BridgeIT RAG-as-a-Service.

What results were reported?

RAG pipeline success rate: 99.982%; Pod restarts: Zero restarts; MTTR for LLM failure cases: reduces mean time to resolution (MTTR) for LLM failure cases; Hallucination risk: Reduce hallucinations (source-reported, not independently verified).

What failed first in this deployment?

How is this quality assurance AI workflow structured?

User query submitted → RAG document retrieval → Source reliability classification → LLM response generation → Structured observability log capture → Dashboard monitoring and alerting → Root cause analysis.