It support · Production

incident.io describes prompt engineering techniques for LLM-driven incident management

The problem

Engineers at incident.io hit persistent walls building LLM-powered incident features: combining classification and generation tasks caused interference, conflicting prompt instructions produced inconsistent results, poorly structured prompts degraded reasoning, and unpreprocessed timestamps caused the model to miscalculate durations.

First attempt

Combining classification and natural language generation in one prompt caused the tasks to interfere; a user-role message phrased as 'this is the message that is related' biased the LLM into over-identifying relevance because LLMs give more attention to recent tokens; and raw timestamps caused incorrect duration calculations.

Workflow diagram · grounded in source
1
Incident alert triggers investigation
trigger
“if an alert comes in about database CPU being high, and someone recently posted a message in #engineering about how to handle situations like this, we'd want this prompt to highlight it”
2
Code change risk classification
ai_action
“the result of a prompt which checks how likely a code change was to cause an incident”
3
Relevant Slack message identification
ai_action
“find Slack discussions relevant to an incident”
4
Structured data preprocessing
integration
“When working with structured data, always ask yourself: How can I make this easier for the LLM?”
5
Prompt health validation
validation
“We use LLM's to check the "health" of our prompts regularly”
Reported outcome

Applying task splitting, instruction deconfliction, structured prompts, preprocessing, and model switching improved results, including one eval suite moving from 50% failures to 100% passing after switching to Sonnet.

Reported metrics
Eval suite pass rate after model switch50% failures to 100% passing
Code change classification performance after preprocessinginstantly improved performance
Reported stack
openaiSonnet4oSlack
Source
https://incident.io/building-with-ai/tricks-to-fix-stubborn-prompts
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Applying task splitting, instruction deconfliction, structured prompts, preprocessing, and model switching improved results, including one eval suite moving from 50% failures to 100% passing after switching to Sonnet.

What tools did this team use?

openai, Sonnet, 4o, Slack.

What results were reported?

Eval suite pass rate after model switch: 50% failures to 100% passing; Code change classification performance after preprocessing: instantly improved performance (source-reported, not independently verified).

What failed first in this deployment?

Combining classification and natural language generation in one prompt caused the tasks to interfere; a user-role message phrased as 'this is the message that is related' biased the LLM into over-identifying relevance…

How is this it support AI workflow structured?

Incident alert triggers investigation → Code change risk classification → Relevant Slack message identification → Structured data preprocessing → Prompt health validation.

Related it support cases