Incident management · Production

incident.io reduces Investigations agent LLM latency 4x through prompt format optimization

The problem

incident.io's Investigations agent LLM prompt calls were slow, taking up to 11 seconds to respond, driven by verbose JSON output with reasoning fields and uncompressed Grafana dashboard definitions that inflated input tokens to about 15k.

First attempt

The initial prompt included reasoning fields that inflated output tokens to 315 and represented Grafana dashboards as verbose JSON, inflating input tokens to about 15k — together driving latency to 11 seconds per call.

Workflow diagram · grounded in source
1
Incident alert received
Trigger
When an incident is declared, the system receives the incoming alert.
▾ source quote
“When an incident is declared, our system analyses the incoming alert and plans which Grafana dashboards to examine”
2
LLM plans dashboard examination
Ai action
The LLM analyzes the alert and determines which Grafana dashboards to examine, including template parameters, panels, and time ranges for each.
▾ source quote
“our system analyses the incoming alert and plans which Grafana dashboards to examine. For each dashboard, we need to determine the right template parameters, specific panels to focus on, and appropriate time ranges”
3
Screenshots pulled from Grafana
Integration
Screenshots are pulled from Grafana for analysis.
▾ source quote
“pulling screenshots from Grafana for analysis”
4
Dashboards ready for responders
Output
Responders can check useful information before they even arrive on the scene.
▾ source quote
“This helps responders check useful information before they even arrive on the scene”
Reported outcome

Through three sequential optimizations — removing reasoning fields, compressing input format, and compressing output format — the Investigations agent prompt went from 11 seconds to reliably under 2.3 seconds, a 4x improvement overall.

Reported metrics
Overall latency improvement4x faster
Output tokens after removing reasoning fields315 to 170
Latency improvement from removing reasoning fields40%
Latency after removing reasoning fields11s to 7s
Show all 10 reported metrics
overall latency improvement4x faster
output tokens after removing reasoning fields315 to 170
latency improvement from removing reasoning fields40%
latency after removing reasoning fields11s to 7s
input token reduction from compressed format>80% reduction in input token usage
latency improvement from input compression20%
latency after input compression7s to 5.7s
latency after output compressionreliably <2.3s
latency reduction from output compression60%
output token reduction from output compression70%
Reported stack
GrafanaGo
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://incident.io/building-with-ai/optimizing-llm-prompts
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Through three sequential optimizations — removing reasoning fields, compressing input format, and compressing output format — the Investigations agent prompt went from 11 seconds to reliably under 2.3 seconds, a 4x im…

What tools did this team use?

Grafana, Go.

What results were reported?

Overall latency improvement: 4x faster; Output tokens after removing reasoning fields: 315 to 170; Latency improvement from removing reasoning fields: 40%; Latency after removing reasoning fields: 11s to 7s (source-reported, not independently verified).

What failed first in this deployment?

The initial prompt included reasoning fields that inflated output tokens to 315 and represented Grafana dashboards as verbose JSON, inflating input tokens to about 15k — together driving latency to 11 seconds per call.

How is this incident management AI workflow structured?

Incident alert received → LLM plans dashboard examination → Screenshots pulled from Grafana → Dashboards ready for responders.

Related incident management cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Incident management cases →