Ticket triage · Production

incident.io builds reliable AI into incident management with evals infrastructure and an AI SRE product

The problem

Building reliable AI features for a reliability-critical product proved far harder than building prototypes — AI's non-determinism meant 'mostly right' was unacceptable when customers make real-world decisions during critical incidents, and scaling efforts caused regressions in one area whenever another improved.

First attempt

Early AI features worked as demos but failed to perform consistently across all customers, datasets, and question phrasings, immediately eroding user trust when failures occurred.

Workflow diagram · grounded in source
1
LLM release triggers exploration
trigger
“OpenAI released GPT-3.5 and ChatGPT, and everything shifted”
2
Prototype AI features built
ai_action
“small prototypes that summarized incidents, suggested actions, or drafted postmortems and executive comms”
3
Reliability gap surfaces
validation
“Getting a model to perform consistently across every situation, every customer, every dataset, every phrasing of a question? It's incredibly hard”
4
Eval and test platform built
feedback_loop
“we built out our foundations: evals, scoring frameworks, backtests, and training sets. Alongside developing our AI features, we created an internal test and experimentation platform that lets us measure, compare models on release, and it…”
5
AI features shipped in platform
output
“automatically generated post-mortems and incident summaries to the ability to ask questions directly in the dashboard and get instant answers about your incidents”
6
AI SRE automates incident response
ai_action
“our new AI SRE product... shows how AI can support reliability engineering by automating some of the hardest and most time-critical parts of incident response”
Reported outcome

incident.io built internal evals, scoring frameworks, backtests, and training sets that made their AI product genuinely dependable, and now ships automatically generated post-mortems, incident summaries, a dashboard Q&A, and a new AI SRE product aimed at substantially reducing downtime and noise.

Reported metrics
Downtime impactsubstantially reduce downtime
Alert noisecut noise
Engineer focus on high-leverage workgive engineers more space to focus on complex, high-leverage problems
Reported stack
Claude CodeSlackScribeInvestigations
Source
https://incident.io/building-with-ai/weaving-ai-into-the-fabric-of-incident-io
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

incident.io built internal evals, scoring frameworks, backtests, and training sets that made their AI product genuinely dependable, and now ships automatically generated post-mortems, incident summaries, a dashboard Q…

What tools did this team use?

Claude Code, Slack, Scribe, Investigations.

What results were reported?

Downtime impact: substantially reduce downtime; Alert noise: cut noise; Engineer focus on high-leverage work: give engineers more space to focus on complex, high-leverage problems (source-reported, not independently verified).

What failed first in this deployment?

Early AI features worked as demos but failed to perform consistently across all customers, datasets, and question phrasings, immediately eroding user trust when failures occurred.

How is this ticket triage AI workflow structured?

LLM release triggers exploration → Prototype AI features built → Reliability gap surfaces → Eval and test platform built → AI features shipped in platform → AI SRE automates incident response.

Related ticket triage cases