Incident management · Production

Mercari builds IBIS, an LLM-powered SRE incident handling buddy using RAG over historical incident reports

The problem

Mercari's on-call SRE team was burdened by frequent alerts escalating into incidents, increasing MTTR and reducing time available for feature development.

Workflow diagram · grounded in source
1
Scheduled incident data export
trigger
“We utilize Google Cloud Scheduler to regularly export the latest incident reports from Blameless's external API into a Google Cloud Storage bucket”
2
Data cleansing
validation
“we leverage the fact that the data is in Markdown format and use the Markdown Splitter function provided by LangChain to extract relevant sections. As for PII, since it has multiple types, we opted to employ the SpaCy NLP model for token…”
3
Translation and summarization
ai_action
“We utilize GPT-4o-based LangChain to handle the translation step. Moreover, since many reports are lengthy, summarizing the content is also crucial for improving vector search precision. GPT-4o assists us in summarizing the data as well”
4
Embedding and vector storage
ai_action
“we utilize OpenAI's embedding model to create vectors from these data sources. The translated and summarized clean data undergoes embedding and is stored in our Vector Database”
5
Engineer queries IBIS in Slack
trigger
“users can directly engage with IBIS through natural language questions by mentioning the bot in Slack”
6
RAG retrieval and response generation
ai_action
“It retrieves relevant information from BigQuery, which acts as our Vector DB, and then sends the data to an LLM model to generate responses. In addition to handling queries, the conversation component also supports other functionalities,…”
7
Incident resolution output
output
“formulates a response to the user by organizing the relevant language constructs”
Reported outcome

IBIS was deployed into several key incident-handling Slack channels at Mercari by end of December 2024, with user adoption continuing to grow while MTTR impact is being monitored.

Reported metrics
Mean Time to Recovery (MTTR)reducing the Mean Time to Recovery (MTTR)
On-call handling costsreducing on-call handling costs
Reported stack
BlamelessGoogle Cloud SchedulerGoogle Cloud StorageGoogle Cloud Run JobsGoogle Cloud Run FunctionsGoogle Cloud WorkflowEventarcLangChainSpaCyGPT-4oOpenAIBigQuerySlack
Source
https://engineering.mercari.com/en/blog/entry/20250206-llm-sre-incident-handling-buddy/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

IBIS was deployed into several key incident-handling Slack channels at Mercari by end of December 2024, with user adoption continuing to grow while MTTR impact is being monitored.

What tools did this team use?

Blameless, Google Cloud Scheduler, Google Cloud Storage, Google Cloud Run Jobs, Google Cloud Run Functions, Google Cloud Workflow, Eventarc, LangChain, SpaCy, GPT-4o.

What results were reported?

Mean Time to Recovery (MTTR): reducing the Mean Time to Recovery (MTTR); On-call handling costs: reducing on-call handling costs (source-reported, not independently verified).

How is this incident management AI workflow structured?

Scheduled incident data export → Data cleansing → Translation and summarization → Embedding and vector storage → Engineer queries IBIS in Slack → RAG retrieval and response generation → Incident resolution output.

Related incident management cases