Incident management · Production

Slack Security Engineering builds multi-agent AI pipeline to streamline on-call security investigations

The problem

Slack's Security Engineering team handles a vast volume of security events from diverse data sources and spends on-call shifts doing the laborious work of manually gathering and reviewing alert evidence.

First attempt

A prompt-based prototype produced highly variable results — sometimes jumping to spurious conclusions without adequately questioning its methods — and prompt refinement alone could not achieve the fine-grained control needed for consistent investigation performance.

Workflow diagram · grounded in source
1
Detection alert triggers investigation
trigger
“allowing investigations to be triggered by our existing detection tools”
2
Director broadcasts discovery questions
ai_action
“The Director reviews the state of the investigation and generates a question that is broadcast to the entire expert team”
3
Expert agents gather domain findings
ai_action
“Each domain expert has a unique set of domain knowledge and data sources. The experts' responsibility is to produce findings from their data sources in response to the Director's questions. We currently have four experts in our team”
4
Critic scores and validates findings
validation
“The Critic's responsibility is to assess and quantify the quality of findings made by domain experts using a rubric we've defined. The Critic annotates the experts' findings with its own analysis and a credibility score for each finding.…”
5
Director routes to next phase
routing
“A "meta-phase" in which the Director decides whether to advance to the next investigation phase or continue in the current one”
6
Interactive investigation report
output
“investigations yield interactive, verifiable reports that show how evidence was collected, interpreted, and judged”
Reported outcome

Slack's on-call team has shifted from manually gathering evidence to supervising AI investigation teams, with agents making spontaneous and unprompted discoveries and yielding interactive, verifiable investigation reports.

Reported metrics
Security events ingested per daybillions
On-call work mode shiftswitching to supervising investigation teams, rather than doing the laborious work of gathering evidence
Agent discovery capabilityspontaneous and unprompted discoveries
Overall benefitmeaningful benefits
Reported stack
coding agent CLI
Source
https://slack.engineering/streamlining-security-investigations-with-agents/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Slack's on-call team has shifted from manually gathering evidence to supervising AI investigation teams, with agents making spontaneous and unprompted discoveries and yielding interactive, verifiable investigation rep…

What tools did this team use?

coding agent CLI.

What results were reported?

Security events ingested per day: billions; On-call work mode shift: switching to supervising investigation teams, rather than doing the laborious work of gathering evidence; Agent discovery capability: spontaneous and unprompted discoveries; Overall benefit: meaningful benefits (source-reported, not independently verified).

What failed first in this deployment?

A prompt-based prototype produced highly variable results — sometimes jumping to spurious conclusions without adequately questioning its methods — and prompt refinement alone could not achieve the fine-grained control…

How is this incident management AI workflow structured?

Detection alert triggers investigation → Director broadcasts discovery questions → Expert agents gather domain findings → Critic scores and validates findings → Director routes to next phase → Interactive investigation report.

Related incident management cases