Ticket triage · Production

incident.io builds Workbench, an internal AI evaluation suite for their incident investigation agent

The problem

As incident.io moved from tightly focused first-generation AI features to a complex AI agent for incident investigation, triage, and resolution, their existing lightweight tooling was insufficient — it lacked eval suites, graders, and scorecards needed to ensure quality at that scale.

First attempt

Off-the-shelf AI tooling options existed but were rejected because relying on vendor marketing rather than first-hand experience risked adopting a product built for a different team context, which would have caused the team to skip learning AI engineering from first principles.

Workflow diagram · grounded in source

@incident interaction trigger

trigger

“When someone interacts with us via @incident, we run some further LLM prompts to classify and score the interaction.”

LLM classification and scoring

ai_action

“we run some further LLM prompts to classify and score the interaction”

Investigations agent analyzes incident

ai_action

“The agent analyzes source code to pinpoint what broke, reaches into Grafana to interpret telemetry data, and connects dots across disparate systems”

Engineer review in Workbench

human_review

“We can then review each interaction in workbench, alongside our classification and scorecard. We also include extra context around the interaction so we can see if the user provided any relevant feedback in the thread (e.g. 'thank-you' o…”

Trace view debugging

validation

“We use this trace view to show us the series of prompts that were run to power a given interaction, enabling us to debug more complex examples: you can easily see what path our interaction took through our prompt tree, and any errors tha…”

Reproducibility testing

validation

“it's useful to be able to re-run it a few times to see if it's an easily reproducible failure case, or an unpredictable one”

AI performance dashboards

feedback_loop

“we can easily build dashboards that join product data with our AI-specific metadata (e.g. scorecards, costs and latency). That helps us do spend analysis and forecasting, as well as being able to report on aggregated scorecards over time”

Reported outcome

incident.io built Workbench, a bespoke internal AI evaluation suite that enabled rapid iteration, a single pane of glass for debugging LLM interactions, and privacy-preserving performance analysis of their Investigations agent without exposing customer data to staff.

Reported metrics

Latency saved via speculative tool callingabout 2s

Idea-to-production cycle timein production by lunchtime

engineer deep-work time on AI improvementspend more time thinking deeply about how to improve the product

Reported stack

WorkbenchLLMGrafanaSonnet 3.7Slack

Source

https://incident.io/building-with-ai/built-our-own-ai-tooling

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

incident.io built Workbench, a bespoke internal AI evaluation suite that enabled rapid iteration, a single pane of glass for debugging LLM interactions, and privacy-preserving performance analysis of their Investigati…

What tools did this team use?

Workbench, LLM, Grafana, Sonnet 3.7, Slack.

What results were reported?

Latency saved via speculative tool calling: about 2s; Idea-to-production cycle time: in production by lunchtime; engineer deep-work time on AI improvement: spend more time thinking deeply about how to improve the product (source-reported, not independently verified).

What failed first in this deployment?

How is this ticket triage AI workflow structured?

@incident interaction trigger → LLM classification and scoring → Investigations agent analyzes incident → Engineer review in Workbench → Trace view debugging → Reproducibility testing → AI performance dashboards.