Customer support · Production

Elastic Field Engineering builds a GenAI customer support chatbot chat interface with RAG and streaming

The problem

Building a chat interface for a GenAI support assistant presented novel UI/UX challenges: users were left waiting with no feedback during slow LLM responses, streaming connections could hang silently for over a minute, and conveying complex multi-source conversation context inside a constrained UI required new design patterns.

First attempt

The first LLM endpoint used for internal alpha-testing did not stream its responses, returning the entire answer in a single HTTP response body, which caused unacceptably long waits. Separately, live streaming connections would frequently return a 200 OK and then hang, with most failed streams taking over a minute to resolve.

Workflow diagram · grounded in source

User submits question

trigger

“the primary function that calls a streaming LLM API after a user asks a question”

RAG search against cluster

ai_action

“RAG search | Server -> cluster | 1 - 2.5s”

Chat history context assembled

integration

“turn all previous chat messages into a JSON object and send it along with the latest question to the LLM endpoint”

LLM called with streaming

ai_action

“Call to LLM | Server -> LLM | 1 - 2.5s”

Killswitch monitors dead stream

validation

“after only 10 seconds of radio silence, we could be nearly certain that the stream would either eventually fail or take longer than one minute to pick back up”

Streamed response delivered to client

output

“First streamed byte | LLM -> server -> client | 3 - 6s”

Reported outcome

The team shipped a custom chat interface using their EUI component library with a branded loading animation, a 10-second killswitch for stalled streams, and a prepended context-selector UI that lets users choose and edit multiple context sources before submitting a question.

Reported metrics

Initial request latency (client to server)100 - 500ms

RAG search latency (server to cluster)1 - 2.5s

LLM call latency (server to LLM)1 - 2.5s

First streamed byte latency3 - 6s

Show all 7 reported metrics

initial request latency (client to server)100 - 500ms

RAG search latency (server to cluster)1 - 2.5s

LLM call latency (server to LLM)1 - 2.5s

first streamed byte latency3 - 6s

total end-to-end latency5.1 - 11.5 seconds

killswitch timeout threshold10 seconds

failed stream resolution timeover one minute

Reported stack

EUIRAGLLM

Source

https://www.elastic.co/search-labs/blog/genai-elastic-elser-chat-interface

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The team shipped a custom chat interface using their EUI component library with a branded loading animation, a 10-second killswitch for stalled streams, and a prepended context-selector UI that lets users choose and e…

What tools did this team use?

EUI, RAG, LLM.

What results were reported?

Initial request latency (client to server): 100 - 500ms; RAG search latency (server to cluster): 1 - 2.5s; LLM call latency (server to LLM): 1 - 2.5s; First streamed byte latency: 3 - 6s (source-reported, not independently verified).

What failed first in this deployment?

The first LLM endpoint used for internal alpha-testing did not stream its responses, returning the entire answer in a single HTTP response body, which caused unacceptably long waits.

How is this customer support AI workflow structured?

User submits question → RAG search against cluster → Chat history context assembled → LLM called with streaming → Killswitch monitors dead stream → Streamed response delivered to client.