incident.io describes prompt engineering techniques for LLM-driven incident management
Engineers at incident.io hit persistent walls building LLM-powered incident features: combining classification and generation tasks caused interference, conflicting prompt instructions produced inconsistent results, poorly structured prompts degraded reasoning, and unpreprocessed timestamps caused the model to miscalculate durations.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Incident alert triggers investigation
An alert (e.g. database CPU high) triggers an investigation that looks for relevant code changes and Slack messages.
Tools used
openaiSonnet4o
Outcome
Applying task splitting, instruction deconfliction, structured prompts, preprocessing, and model switching improved results, including one eval suite moving from 50% failures to 100% passing after switching to Sonnet.
What failed first
Combining classification and natural language generation in one prompt caused the tasks to interfere; a user-role message phrased as 'this is the message that is related' biased the LLM into over-identifying relevance because LLMs give more attention to recent tokens; and raw timestamps caused incorrect duration calculations.