Incident management · Production

Canva auto-generates Post Incident Review summaries with GPT-4-chat

The problem

Canva's reliability engineers manually wrote Post Incident Review summaries after every incident, but over time the summaries became inconsistent and reviewers often lacked the context needed to review them quickly and effectively, creating ongoing toil for the engineering team.

First attempt

A fine-tuned GPT model was evaluated as a candidate approach but discarded because the available training examples were insufficient to produce summaries capturing the specific details needed, and manual comparison showed the fine-tuned model underperformed GPT completion and GPT chat on accurately determining impact duration and correlating incident phases.

Workflow diagram · grounded in source

Fetch PIR from Confluence

trigger

“We start by fetching the report from Confluence and parsing the HTML to extract the content of the PIR as raw text”

Remove sensitive data

validation

“we remove sensitive data, including links, emails, and Slack channel names, to avoid exposing internal information to public models and ensure blameless summaries”

GPT-4 generates summary

ai_action

“We then send the text version of the report to GPT-4 chat completion to generate a summary”

Archive in data warehouse

integration

“we archive it in our data warehouse, allowing us to integrate it with additional incident metadata, which we store for comprehensive reporting purposes”

Post summary to Jira

output

“We also include the summary on the Jira tickets for the incident, so any manual changes are recorded to the warehouse using Jira webhooks”

Compare AI vs human edits

feedback_loop

“This allows us to compare the AI-generated and human-modified summaries to monitor the data quality”

Reported outcome

After approximately two months in production, most AI-generated PIR summaries remain unaltered by engineers, demonstrating the team's approval of GPT-4's output quality, with the process significantly improving the efficiency and consistency of PIR summarization and reducing operational toil for reliability engineers.

Reported metrics

PIR summaries unaltered by engineersmost of the AI-generated PIR summaries remain unaltered

efficiency and consistency of PIR summarizationsignificantly improved the efficiency and consistency

Operational toil for reliability engineersreduced the operational toil

Maximum estimated cost per summary$0.6

Show all 5 reported metrics

PIR summaries unaltered by engineersmost of the AI-generated PIR summaries remain unaltered

efficiency and consistency of PIR summarizationsignificantly improved the efficiency and consistency

operational toil for reliability engineersreduced the operational toil

maximum estimated cost per summary$0.6

GPT-4 cost per 1K tokens$0.06

Reported stack

GPT-4-chatConfluenceJiradata warehouse

Source

https://www.canva.dev/blog/engineering/summarise-post-incident-reviews-with-gpt4/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After approximately two months in production, most AI-generated PIR summaries remain unaltered by engineers, demonstrating the team's approval of GPT-4's output quality, with the process significantly improving the ef…

What tools did this team use?

GPT-4-chat, Confluence, Jira, data warehouse.

What results were reported?

PIR summaries unaltered by engineers: most of the AI-generated PIR summaries remain unaltered; efficiency and consistency of PIR summarization: significantly improved the efficiency and consistency; Operational toil for reliability engineers: reduced the operational toil; Maximum estimated cost per summary: $0.6 (source-reported, not independently verified).

What failed first in this deployment?

How is this incident management AI workflow structured?

Fetch PIR from Confluence → Remove sensitive data → GPT-4 generates summary → Archive in data warehouse → Post summary to Jira → Compare AI vs human edits.