Incident management · Production

Trendyol builds an AI multi-agent oncall system that diagnoses production alerts in minutes

The problem

Trendyol's oncall engineers spent 30–60 minutes on manual investigation per production alert — checking logs, metrics dashboards, code, and infrastructure across multiple microservices — while the actual fix was often trivial.

Workflow diagram · grounded in source
1
Alert received in Slack
trigger
“An alert arrives in a Slack channel Our AI system detects it and begins investigation”
2
Knowledge base fast-path
validation
“If the alert message contains a known error pattern, the solution is returned instantly — no agents are launched, no metrics are queried, no code is analyzed.”
3
Alert classification
ai_action
“it classifies the alert into one of four categories”
4
Service discovery and topology retrieval
integration
“Once the service is identified, the Coordinator fetches its service topology — a map of all its dependencies”
5
Parallel specialist agent dispatch
ai_action
“All applicable agents run simultaneously. The Alert Analyzer starts at the same time as the PostgreSQL and Kafka agents, not after them.”
6
Smart result aggregation
ai_action
“When results come back, the Coordinator applies smart aggregation logic — and the aggregation path differs by classification”
7
Root cause analysis posted to Slack
output
“A structured root cause analysis with probability-ranked hypotheses is posted back to Slack”
8
Conversational follow-up investigation
human_review
“Engineers can now mention the bot in Slack to continue the investigation in real-time. No laptop required, no dashboards to open — just tag the bot and ask.”
Reported outcome

Since deploying the Oncall Support Workspace, investigation time dropped from 30–60 minutes to minutes, oncall engineers report less stress and faster context acquisition, and known false positives are resolved instantly without waking anyone up.

Reported metrics
investigation time (MTTR)from 30–60 minutes of manual investigation to structured root cause analysis in minutes
Knowledge base fast-path resolution timeunder a second
Scenario alert-to-diagnosis timeunder 5 minutes
Reported stack
ElasticsearchPostgreSQLKafkaCouchbaseSlack
Source
https://medium.com/trendyol-tech/how-we-built-an-ai-powered-oncall-system-that-diagnoses-production-alerts-in-minutes-86386be0d4b8
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Since deploying the Oncall Support Workspace, investigation time dropped from 30–60 minutes to minutes, oncall engineers report less stress and faster context acquisition, and known false positives are resolved instan…

What tools did this team use?

Elasticsearch, PostgreSQL, Kafka, Couchbase, Slack.

What results were reported?

investigation time (MTTR): from 30–60 minutes of manual investigation to structured root cause analysis in minutes; Knowledge base fast-path resolution time: under a second; Scenario alert-to-diagnosis time: under 5 minutes (source-reported, not independently verified).

How is this incident management AI workflow structured?

Alert received in Slack → Knowledge base fast-path → Alert classification → Service discovery and topology retrieval → Parallel specialist agent dispatch → Smart result aggregation → Root cause analysis posted to Slack → Conversational follow-up investigation.

Related incident management cases