It support · Production

incident.io reduces chatbot latency by 50% with speculative tool calling

The problem

incident.io's incident-management chatbot suffered multi-second delays because LLM tool calls require sequential round-trips — the model decides to call a tool, the tool runs, the result returns, and only then does the model continue — producing nearly 5 seconds of wait time for a simple 'pause incident' action.

First attempt

Prompt optimization alone was insufficient to meet latency targets; all available LLM-level and database-level performance improvements had already been applied before the speculative approach was developed.

Workflow diagram · grounded in source
1
User sends incident request
trigger
“you type @incident pause until tomorrow”
2
Keyword-based intent routing
routing
“we use a list of keywords to decide when we should speculatively call a tool. That's working really well for us: we catch almost all the cases where we do want to call the tool, and the false-positive rate is low enough that our costs ar…”
3
Speculative tool call launched
integration
“we call ToolIncidentUpdate, just in case it's needed. And if we decide not to use it, we can just cancel the request”
4
LLM evaluates and decides
ai_action
“PromptChat, which identifies that it needs to make a change to the incident so it calls a tool ToolIncidentUpdate”
5
WriteBarrier gates actual writes
validation
“we implement this as a WriteBarrier. When we call a tool call 'speculatively', we pass down a write barrier that blocks writes (in our case, actually updating the incident) while we figure out whether the call is genuine”
6
Update committed
integration
“if PromptChat concludes that it does want to call ToolIncidentUpdate, we unblock the barrier, and let our tool continue to run”
7
Optimistic confirmation drafted
output
“calling PromptChat optimistically: assuming that we have successfully updated the incident”
Reported outcome

Speculative tool calling saves 2–3 seconds per interaction and reduces latency by about 50% for most users, making the chatbot feel responsive rather than sluggish.

Reported metrics
Latency savings per interaction2-3s
Latency reduction for most usersabout 50%
Latency savings in example scenario1.6s
Baseline time to pause incident (pre-optimization)nearly 5 seconds
Reported stack
LLMGo
Source
https://incident.io/building-with-ai/speculative-tool-calling
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Speculative tool calling saves 2–3 seconds per interaction and reduces latency by about 50% for most users, making the chatbot feel responsive rather than sluggish.

What tools did this team use?

LLM, Go.

What results were reported?

Latency savings per interaction: 2-3s; Latency reduction for most users: about 50%; Latency savings in example scenario: 1.6s; Baseline time to pause incident (pre-optimization): nearly 5 seconds (source-reported, not independently verified).

What failed first in this deployment?

Prompt optimization alone was insufficient to meet latency targets; all available LLM-level and database-level performance improvements had already been applied before the speculative approach was developed.

How is this it support AI workflow structured?

User sends incident request → Keyword-based intent routing → Speculative tool call launched → LLM evaluates and decides → WriteBarrier gates actual writes → Update committed → Optimistic confirmation drafted.

Related it support cases