Incident management · Production

Zalando builds AI-powered multi-stage LLM pipeline to transform two years of postmortems into actionable infrastructure insights

The problem

Zalando accumulated thousands of postmortem documents but could not extract strategic patterns at scale. Each postmortem takes 15–20 minutes to read, making company-wide retrospective analysis of years of incidents cognitively and practically impossible.

First attempt

An initial attempt using Google's NotebookLM produced severe hallucinations and lost incident context when generating summaries, reducing effective productivity rather than improving it. Small open-source models showed up to 40% hallucination probability, and a no-code agentic approach was ruled out due to performance limitations and inaccuracies.

Workflow diagram · grounded in source
1
Postmortem corpus ingested
trigger
“Pipeline's input is thousands of postmortem documents”
2
LLM summarization
ai_action
“Using a tightly scoped prompt, we have used Turn, Expression, Level of Details, Role (TELeR) techniques for prompt engineering, LLM processes each postmortem document and extracts only the most essential information across five core dime…”
3
Technology classification
ai_action
“The stage systematically identifies whether specific datastore technologies directly contributed to the incident. The process works as follows: the model receives a summary postmortem document along with a list of technologies in question.”
4
Incident digest extraction
ai_action
“The most crucial part of the incident analysis is the extraction of a short 3 to 5 sentence digest that highlights (a) the root cause or fault condition involving the technology; (b) the role it played in the overall failure scenario; (c…”
5
Human curation
human_review
“During the pipeline development, we conducted 100% human curation of output batches. This involved analyzing the generated postmortem digests and comparing them to the original postmortems. The curation process was purely labelling, requ…”
6
Cross-incident pattern detection
ai_action
“We are feeding the entire set of incident digests into LLM within a single prompt. Within the prompt, we are explicitly prohibiting inference, redundancy, or the inclusion of any information not grounded in the source data. The output is…”
7
Investment opportunity report
output
“the output is a one-pager describing the trends and patterns for incidents in the focus”
Reported outcome

The multi-stage LLM pipeline reduced postmortem analysis time from days to hours and boosted productivity three times.
It surfaced hidden patterns including a finding that automated change validation could shield 25% of subsequent datastore incidents. Surface attribution error remains at approximately 10% even with the latest model, and hallucinations became negligible.

Reported metrics
Postmortem analysis timesignificantly reduced the time for analysis from days to hours
Productivity boostthree times
Summary reading time5 minutes
Manual postmortem reading time15-20 minutes
Show all 14 reported metrics
postmortem analysis timesignificantly reduced the time for analysis from days to hours
productivity boostthree times
summary reading time5 minutes
manual postmortem reading time15-20 minutes
subsequent datastore incidents shielded by automated change validation25%
AWS ElastiCache CPU utilization at peak traffic80%
hallucination probability with small modelsup to 40%
hallucination rate after human curationless than 15%
hallucination rate with large modelnegligible
surface attribution error rate with Claude Sonnet 4approximately 10%
processing time per postmortem (Claude Sonnet 4)approximately 30 seconds
annual data processing timeunder 24 hours
human curation rate (initial)100%
human curation rate (mature system)10-20% of randomly sampled summaries
Reported stack
Claude Sonnet 4AWS BedrockNotebookLMLM Studio
Source
https://engineering.zalando.com/posts/2025/09/dead-ends-or-data-goldmines-ai-powered-postmortem-analysis.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The multi-stage LLM pipeline reduced postmortem analysis time from days to hours and boosted productivity three times.

What tools did this team use?

Claude Sonnet 4, AWS Bedrock, NotebookLM, LM Studio.

What results were reported?

Postmortem analysis time: significantly reduced the time for analysis from days to hours; Productivity boost: three times; Summary reading time: 5 minutes; Manual postmortem reading time: 15-20 minutes (source-reported, not independently verified).

What failed first in this deployment?

An initial attempt using Google's NotebookLM produced severe hallucinations and lost incident context when generating summaries, reducing effective productivity rather than improving it.

How is this incident management AI workflow structured?

Postmortem corpus ingested → LLM summarization → Technology classification → Incident digest extraction → Human curation → Cross-incident pattern detection → Investment opportunity report.

Related incident management cases