Back office ops · Production

How Shortwave built an AI email executive assistant using RAG

The problem

Shortwave's attempts to turn users' email history into an actionable knowledge base using traditional search infrastructure were underwhelming, and the team wanted to unlock the full potential of LLMs to change how users interact with their inbox.

First attempt

Traditional search infrastructure fell short for email knowledge retrieval, and multi-stage LLM call chain architectures proved impractical because they introduced data loss and errors at each stage, leading to low-quality responses.

Workflow diagram · grounded in source
1
User query received
trigger
“When you ask the Shortwave AI a question, it immediately passes that query to an LLM”
2
Tool selection
ai_action
“We use GPT-4 for this step, and we include a lot of detail about the state of the world in our prompt, such as current app state (ie. the current thread the user is viewing), what tools are available, conversation history, and more”
3
Parallel tool data retrieval
integration
“we have all of the tools retrieve their data in parallel”
4
Query reformulation
ai_action
“Query reformulation solves this problem. It takes a query that lacks needed context, and rewrites it using an LLM so that it makes sense on its own. Query reformulation considers anything that is visible on the screen that the user might…”
5
Parallel feature extraction
ai_action
“We do this via a large number of parallel calls to a fast LLM, each optimized to extract a specific feature. Having these LLM calls be independent of each other enables a degree of modularity that makes it easier to test and evolve them.…”
6
Semantic embedding search
ai_action
“we embed the user's reformulated query and search our vector database for semantically similar emails”
7
Heuristics-based re-ranking
validation
“the first phase uses fast heuristics to very quickly cut the list down from a large number to something more manageable for our next machine learning based re-ranking phase”
8
Cross-encoder re-ranking
ai_action
“Our most powerful technique for re-ranking is an open source cross encoding model purpose-built for this task. We use MS Marco MiniLM, running on our own GPUs”
9
Answer generation
ai_action
“create a prompt containing the original user question and all the context information fetched using various tools, and pass it along to an LLM (GPT-4 in our case) to produce an answer”
10
Post-processing and output
output
“Our post-processing step converts the LLM's output to rich text, adds source citations, and suggests actions to the user (ie. "Insert draft")”
Reported outcome

Shortwave's AI assistant answers almost all questions within 3-5 seconds end-to-end across multiple LLM calls, vector DB lookups, and ML model inference steps.

Reported metrics
End-to-end response latency3-5 seconds
Reported stack
GPT-4InstructorPineconeMS Marco MiniLMElasticSearch
Source
https://www.shortwave.com/blog/deep-dive-into-worlds-smartest-email-ai/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Shortwave's AI assistant answers almost all questions within 3-5 seconds end-to-end across multiple LLM calls, vector DB lookups, and ML model inference steps.

What tools did this team use?

GPT-4, Instructor, Pinecone, MS Marco MiniLM, ElasticSearch.

What results were reported?

End-to-end response latency: 3-5 seconds (source-reported, not independently verified).

What failed first in this deployment?

Traditional search infrastructure fell short for email knowledge retrieval, and multi-stage LLM call chain architectures proved impractical because they introduced data loss and errors at each stage, leading to low-qu…

How is this back office ops AI workflow structured?

User query received → Tool selection → Parallel tool data retrieval → Query reformulation → Parallel feature extraction → Semantic embedding search → Heuristics-based re-ranking → Cross-encoder re-ranking → Answer generation → Post-processing and output.

Related back office ops cases