Incident management · Production

incident.io reduces Investigations agent LLM latency 4x through prompt format optimization

The problem

incident.io's Investigations agent LLM prompt calls were slow, taking up to 11 seconds to respond, driven by verbose JSON output with reasoning fields and uncompressed Grafana dashboard definitions that inflated input tokens to about 15k.

First attempt

The initial prompt included reasoning fields that inflated output tokens to 315 and represented Grafana dashboards as verbose JSON, inflating input tokens to about 15k — together driving latency to 11 seconds per call.

Workflow diagram · grounded in source
1
Incident alert received
trigger
“When an incident is declared, our system analyses the incoming alert and plans which Grafana dashboards to examine”
2
LLM plans dashboard examination
ai_action
“our system analyses the incoming alert and plans which Grafana dashboards to examine. For each dashboard, we need to determine the right template parameters, specific panels to focus on, and appropriate time ranges”
3
Screenshots pulled from Grafana
integration
“pulling screenshots from Grafana for analysis”
4
Dashboards ready for responders
output
“This helps responders check useful information before they even arrive on the scene”
Reported outcome

Through three sequential optimizations — removing reasoning fields, compressing input format, and compressing output format — the Investigations agent prompt went from 11 seconds to reliably under 2.3 seconds, a 4x improvement overall.

Reported metrics
Overall latency improvement4x faster
Output tokens after removing reasoning fields315 to 170
Latency improvement from removing reasoning fields40%
Latency after removing reasoning fields11s to 7s
Show all 10 reported metrics
overall latency improvement4x faster
output tokens after removing reasoning fields315 to 170
latency improvement from removing reasoning fields40%
latency after removing reasoning fields11s to 7s
input token reduction from compressed format>80% reduction in input token usage
latency improvement from input compression20%
latency after input compression7s to 5.7s
latency after output compressionreliably <2.3s
latency reduction from output compression60%
output token reduction from output compression70%
Reported stack
GrafanaGo
Source
https://incident.io/building-with-ai/optimizing-llm-prompts
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Through three sequential optimizations — removing reasoning fields, compressing input format, and compressing output format — the Investigations agent prompt went from 11 seconds to reliably under 2.3 seconds, a 4x im…

What tools did this team use?

Grafana, Go.

What results were reported?

Overall latency improvement: 4x faster; Output tokens after removing reasoning fields: 315 to 170; Latency improvement from removing reasoning fields: 40%; Latency after removing reasoning fields: 11s to 7s (source-reported, not independently verified).

What failed first in this deployment?

The initial prompt included reasoning fields that inflated output tokens to 315 and represented Grafana dashboards as verbose JSON, inflating input tokens to about 15k — together driving latency to 11 seconds per call.

How is this incident management AI workflow structured?

Incident alert received → LLM plans dashboard examination → Screenshots pulled from Grafana → Dashboards ready for responders.

Related incident management cases