Customer support · Production

GEICO experimental RAG implementation with RagRails hallucination guardrails for conversational quoting

The problem

LLMs used in GEICO's hackathon-winning conversational chat application produced unreliable and inconsistent outputs — including hallucinations and a specific pattern called 'overpromising' where the model incorrectly assumed capabilities it did not have — making the experience unsuitable for public-facing customer interactions.

First attempt

The first RAG implementation placed entire records in the system prompt, producing ineffective and unreliable results; subsequent attempts to permanently fix overpromising by adding instructions directly to the system prompt also failed and disrupted other pipeline goals.

Workflow diagram · grounded in source
1
Offline document vectorization
integration
“splitting documents, converting each to embeddings through an API, then extracting metadata using LLMs. Once the records are split, they are sent along with the metadata and their vector to the database to be indexed”
2
User message triggers pipeline
trigger
“For every interaction, a part of the conversation, the task description, and constraints are dynamically composed based on the quote process stage and the user intention”
3
LLM input translation
ai_action
“every user message was sent to a LLM to translate the input into a form that could be matched within the knowledge base. This translated input was a coherent sentence that tried to predict the question the user was likely to ask, seemed …”
4
Vector database retrieval
integration
“accurate retrieval depended on the semantic closeness of the vectorized representation to the user input”
5
Relevance check
validation
“We used the same LLM to evaluate the records' relevance to a part of the conversation”
6
Result ranking and reordering
ai_action
“By reordering retrieved knowledge and prioritizing the most relevant ones at the beginning or end of the sequence, the LLM's focus window improved, leading to more coherent and on-topic generation”
7
RagRails guardrail injection
ai_action
“RagRails involves adding guiding instructions to a record to aid the LLM. The additional context helps the model to avoid misconceptions and potential negative effects while reinforcing desired behaviors”
8
RAG-augmented response
output
“Incorporating relevant information equips the LLM to provide answers that better align with more stringent expectations for consistency, reliability, and compliance”
Reported outcome

The RagRails strategy reduced overpromising errors from 12 out of 20 test responses to 6 and eventually to none after further adjustments, providing a repeatable mechanism for hallucination control in the RAG pipeline.

Reported metrics
error rate in repeated testing before RagRails12 out of 20 responses were incorrect
errors after initial RagRails adjustment6
errors after further RagRails adjustmentsnone
Reported stack
GPT modelHNSWvector database
Source
https://www.geico.com/techblog/application-of-retrieval-augmented-generation/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The RagRails strategy reduced overpromising errors from 12 out of 20 test responses to 6 and eventually to none after further adjustments, providing a repeatable mechanism for hallucination control in the RAG pipeline.

What tools did this team use?

GPT model, HNSW, vector database.

What results were reported?

error rate in repeated testing before RagRails: 12 out of 20 responses were incorrect; errors after initial RagRails adjustment: 6; errors after further RagRails adjustments: none (source-reported, not independently verified).

What failed first in this deployment?

The first RAG implementation placed entire records in the system prompt, producing ineffective and unreliable results; subsequent attempts to permanently fix overpromising by adding instructions directly to the system…

How is this customer support AI workflow structured?

Offline document vectorization → User message triggers pipeline → LLM input translation → Vector database retrieval → Relevance check → Result ranking and reordering → RagRails guardrail injection → RAG-augmented response.

Related customer support cases