Customer support · Production

Elastic builds a RAG-based knowledge library to power the Elastic Support Assistant

The problem

Elastic's LLM foundational training was insufficient for technically deep product questions; the knowledge base was split across Swiftype and Appsearch instances creating tech debt; and most ingested documents lacked the summaries needed for effective semantic search.

First attempt

Fine-tuning a custom model was explored and rejected because it required question-answer pairing that did not match the existing data set. Using the first 280 characters of each document as a summary led to poor search relevancy. Passing larger text passages as context to the LLM decreased accuracy.

Workflow diagram · grounded in source
1
Support articles ingested into Elasticsearch
integration
“This is run entirely on Elasticsearch on the back end and the EUI Markdown Editor control on the front end and is one of the key information sources for the Elastic Support Assistant.”
2
Crawlee ingests product docs and blogs
integration
“We elected to use a scraping and automation library called Crawlee in order to handle the scale and frequency needed to keep our knowledge library up to date. Each of the four crawler jobs executes on Google Cloud Run.”
3
ELSER enriches documents with embeddings
ai_action
“ELSER enriches Elastic documents by transforming them into enriched embeddings that enhance search relevance and accuracy. This advanced embedding mechanism leverages machine learning to understand the contextual relationships within the…”
4
GPT backfills summaries and questions
ai_action
“We created a new service which leveraged OpenAI GPT3.5 Turbo to backfill all of our documents which lacked a summary upon ingestion. The loop does the following tasks before moving onto the next document. Sends an API call to the LLM wit…”
5
User asks question to Support Assistant
trigger
“When asked a question, the librarian might be able to answer from their own knowledge but may need to find the appropriate book(s) to address questions about deep domain knowledge.”
6
Hybrid search retrieves relevant context
ai_action
“For our hybrid search approach in the Support Assistant, we use the title and summary fields for semantic search with BM25 on the much larger content field. This enables the Support Assistant to have both speed and high relevance to the …”
7
LLM synthesizes answer for user
output
“By extracting this information and sending it to the large language model, we both save the user time and return an answer in natural language that is easy to understand”
Reported outcome

Elastic built a knowledge library with vector embeddings for more than 300,000 documents and over 128,000 AI-generated summaries averaging 8 questions each, representing a 10x improvement for semantic search results and enabling the Support Assistant to answer a much broader range of questions.

Reported metrics
Documents in knowledge library with vector embeddingsmore than 300,000
AI-generated summariesover 128,000
AI-generated questions per documentaverage of 8
Semantic search improvement10x improvement
Show all 8 reported metrics
documents in knowledge library with vector embeddingsmore than 300,000
AI-generated summariesover 128,000
AI-generated questions per documentaverage of 8
semantic search improvement10x improvement
technical support articles in knowledge baseover 2,800
product documentation versions in knowledge library114 unique versions
technical support articles with human-written summaries~8,000
LLM response determinismsignificantly more deterministic
Reported stack
ElasticsearchELSEROpenAI GPT3.5 TurboCrawleeGoogle Cloud RunEUI Markdown EditorBM25
Source
https://www.elastic.co/search-labs/blog/genai-customer-support-building-a-knowledge-library
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Elastic built a knowledge library with vector embeddings for more than 300,000 documents and over 128,000 AI-generated summaries averaging 8 questions each, representing a 10x improvement for semantic search results a…

What tools did this team use?

Elasticsearch, ELSER, OpenAI GPT3.5 Turbo, Crawlee, Google Cloud Run, EUI Markdown Editor, BM25.

What results were reported?

Documents in knowledge library with vector embeddings: more than 300,000; AI-generated summaries: over 128,000; AI-generated questions per document: average of 8; Semantic search improvement: 10x improvement (source-reported, not independently verified).

What failed first in this deployment?

Fine-tuning a custom model was explored and rejected because it required question-answer pairing that did not match the existing data set.

How is this customer support AI workflow structured?

Support articles ingested into Elasticsearch → Crawlee ingests product docs and blogs → ELSER enriches documents with embeddings → GPT backfills summaries and questions → User asks question to Support Assistant → Hybrid search retrieves relevant context → LLM synthesizes answer for user.

Related customer support cases