Customer support · Production

Elastic Field Engineering tunes RAG search for ~75% relevance improvement in Technical Support AI Assistant

The problem

The initial RAG-powered Technical Support AI Assistant returned irrelevant results for CVE and product-version queries, causing the LLM to generate uninformed answers and eroding user trust.

First attempt

Three root causes degraded search quality: document summaries were just the first few characters of the body, misleading semantic embeddings; multiple versions of the same article cluttered top results; and version-specific boosting was absent so wrong article versions were returned.

Workflow diagram · grounded in source
1
Knowledge base ingestion
integration
“our journey began with an extensive database that included over 300,000 documents consisting of Technical Support Knowledge Articles and various pages crawled from our website, such as Elastic's Product Documentation and Blogs”
2
AI document enrichment
ai_action
“leveraging the use of GenAI and the GPT4 model, we made a team decision to craft a new AI Enrichment Service”
3
Hybrid search with boosting
ai_action
“We adopted a standard Hybrid-Search strategy, which combines the traditional strengths of BM25, Keyword-based Search, with the capabilities of Semantic Search, powered by ELSER”
4
LLM response generation
ai_action
“we build up the system prompt with different sets of instructions, also contemplating the top 3 search results as context to be used. Finally, we feed the conversation alongside the built context into the LLM, generating a response”
5
User feedback capture
feedback_loop
“We used a third-party tool to capture client-side events, connecting to Big Query for storage and making the JSON-encoded events accessible for comprehensive analysis by everyone on the team”
6
Relevance evaluation with P@K
validation
“we have employed Elasticsearch's Ranking Evaluation API alongside with the Precision at K (P@K) metric to determine how many relevant results are returned between the first K hits of a query”
Reported outcome

Search relevance improved by ~75% in top-3 results (precisely +78.41% average P@K improvement) and over 300,000 AI-generated summaries were produced for use across future applications.

Reported metrics
Top-3 results relevance increase~75%
average P@K improvement+78.41%
AI-generated summaries createdover 300,000
Reported stack
ElasticsearchELSERGPT4TypeScript/Node.jsRanking Evaluation APIAzure OpenAI
Source
https://www.elastic.co/search-labs/blog/elser-rag-search-for-relevance
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Search relevance improved by ~75% in top-3 results (precisely +78.41% average P@K improvement) and over 300,000 AI-generated summaries were produced for use across future applications.

What tools did this team use?

Elasticsearch, ELSER, GPT4, TypeScript/Node.js, Ranking Evaluation API, Azure OpenAI.

What results were reported?

Top-3 results relevance increase: ~75%; average P@K improvement: +78.41%; AI-generated summaries created: over 300,000 (source-reported, not independently verified).

What failed first in this deployment?

Three root causes degraded search quality: document summaries were just the first few characters of the body, misleading semantic embeddings; multiple versions of the same article cluttered top results; and version-sp…

How is this customer support AI workflow structured?

Knowledge base ingestion → AI document enrichment → Hybrid search with boosting → LLM response generation → User feedback capture → Relevance evaluation with P@K.

Related customer support cases