Back office ops · Production

Google introduces conversation summaries in Google Chat Spaces using the Pegasus abstractive summarization model

The problem

Information overload from the volume of incoming chat messages and documents is a significant challenge for organizations and individuals, worsened by the shift to virtual and hybrid work environments.

First attempt

The abstractive summarization model occasionally produces low-quality outputs of two types: misattribution (confusing who said or did what) and misrepresentation (summaries that contradict the actual conversation). The initial hybrid model also had noticeable latency when users opened Spaces.

Workflow diagram · grounded in source
1
Message event triggers summary update
trigger
“we instead generate and update summaries whenever there is a new message sent, edited or deleted”
2
Pegasus generates abstractive summary
ai_action
“This feature is enabled by our state-of-the-art abstractive summarization model, Pegasus, which generates useful and concise summaries for chat conversations”
3
Summaries cached ephemerally
integration
“summaries are cached ephemerally to ensure they surface smoothly when users open Spaces with unread messages”
4
Summary card shown to user
output
“a card with automatically generated summaries is shown as users enter Spaces with unread messages. The card includes a list of summaries for the different topics discussed in Spaces”
Reported outcome

Google deployed conversation summaries in Google Chat Spaces, showing automatically generated digest cards when users enter Spaces with unread messages; latency was resolved by pre-generating summaries on message events and caching them ephemerally.

Reported metrics
ForumSum training conversations collectedover six thousand
Average speakers per training conversationmore than 6
Average utterances per training conversation10
Distilled model latency and memory footprintlower latency and memory footprint while maintaining similar quality
Show all 5 reported metrics
ForumSum training conversations collectedover six thousand
average speakers per training conversationmore than 6
average utterances per training conversation10
distilled model latency and memory footprintlower latency and memory footprint while maintaining similar quality
summary accuracygenerates useful and accurate summaries most of the time
Reported stack
PegasusForumSum
Source
https://ai.googleblog.com/2022/11/conversation-summaries-in-google-chat.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Google deployed conversation summaries in Google Chat Spaces, showing automatically generated digest cards when users enter Spaces with unread messages; latency was resolved by pre-generating summaries on message even…

What tools did this team use?

Pegasus, ForumSum.

What results were reported?

ForumSum training conversations collected: over six thousand; Average speakers per training conversation: more than 6; Average utterances per training conversation: 10; Distilled model latency and memory footprint: lower latency and memory footprint while maintaining similar quality (source-reported, not independently verified).

What failed first in this deployment?

The abstractive summarization model occasionally produces low-quality outputs of two types: misattribution (confusing who said or did what) and misrepresentation (summaries that contradict the actual conversation).

How is this back office ops AI workflow structured?

Message event triggers summary update → Pegasus generates abstractive summary → Summaries cached ephemerally → Summary card shown to user.

Related back office ops cases