Customer support · Production

Elastic builds observability for its GenAI Support Assistant chatbot

The problem

Elastic's Field Engineering team needed comprehensive observability infrastructure for the newly launched GenAI Support Assistant to detect runtime bugs, monitor latency and usage, and prevent abuse of the LLM service.

First attempt

The initial first-generation timeout was configured on the client side but the server never became aware when the client aborted the request, requiring a redesign to the server-side API layer. A data-loading bug drove endpoint throughput well over 100 transactions per minute, and HTTP 413 errors appeared when RAG context combined with user input exceeded the server's configured payload size limit.

Workflow diagram · grounded in source

User submits support question

trigger

“The goal of the Support Assistant is to be helpful, but there are two broad categories of input that we want to avoid engaging with. The first is questions unrelated to getting technical support for Elastic products.”

RAG search retrieves context

ai_action

“the amount of context generated from our RAG search, combined with the user's input”

LLM generates streaming response

ai_action

“we chose to use streaming responses from the LLM in order to avoid having to wait for the LLM generation to finish before being able to show anything to the user”

First-generation timeout enforcement

validation

“we chose to enforce a 10 second timeout on getting the first chunk of generated text back”

Declined request detection

validation

“by having a predefined response for declining requests, it created an opportunity for me to examine the response coming from the LLM, and compare it to the variable containing the standardized decline message”

Response streamed to client

output

“The streaming responses work by sending a series of events related to the generation down to the client. There are 4 flavors; Started, Generation, End, and Error.”

APM observability feedback loop

feedback_loop

“It's how we noticed a bug causing us to load the same data over and over again from our server. We saw in the APM data that the throughput for one of our endpoints was well over 100 transactions per minute”

Reported outcome

Observability confirmed fixes for runtime bugs, with endpoint throughput dropping to 1 TPM after the data-loading fix, and the Support Assistant served its 100th chat completion 21 hours post-launch with no rate limit violations in the first weeks since launch.

Reported metrics

100th chat completion milestone21 hours post launch

Endpoint throughput after bug fix1 TPM

Average chat completion latency20 seconds

Heavy internal user daily chat volume10-20 chat messages per day on average

Show all 7 reported metrics

100th chat completion milestone21 hours post launch

endpoint throughput after bug fix1 TPM

average chat completion latency20 seconds

heavy internal user daily chat volume10-20 chat messages per day on average

peak internal user single-day chat volumeover 70 in a single day

chat completion rate limit per user per hour20 chat completions

theoretical max chats per user per hour (single tab)180 chats in an hour

Reported stack

Elastic APMFilebeatElasticsearchKibanaElastic Synthetics MonitoringES|QLAzure OpenAIElastic AgentGPT-4GPT-4oLLMRAGElastic CloudSalesforce

Source

https://www.elastic.co/search-labs/blog/genai-customer-support-observability

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Observability confirmed fixes for runtime bugs, with endpoint throughput dropping to 1 TPM after the data-loading fix, and the Support Assistant served its 100th chat completion 21 hours post-launch with no rate limit…

What tools did this team use?

Elastic APM, Filebeat, Elasticsearch, Kibana, Elastic Synthetics Monitoring, ES|QL, Azure OpenAI, Elastic Agent, GPT-4, GPT-4o.

What results were reported?

100th chat completion milestone: 21 hours post launch; Endpoint throughput after bug fix: 1 TPM; Average chat completion latency: 20 seconds; Heavy internal user daily chat volume: 10-20 chat messages per day on average (source-reported, not independently verified).

What failed first in this deployment?

The initial first-generation timeout was configured on the client side but the server never became aware when the client aborted the request, requiring a redesign to the server-side API layer.

How is this customer support AI workflow structured?

User submits support question → RAG search retrieves context → LLM generates streaming response → First-generation timeout enforcement → Declined request detection → Response streamed to client → APM observability feedback loop.