Back office ops · Production

Superhuman's Ask AI cuts email search time by 14% with a multi-agent RAG architecture

The problem

Users spent up to 35 minutes per week struggling with keyword-based email and calendar search, forced to recall exact phrases and sender names with no semantic understanding of their queries.

First attempt

The initial single-prompt LLM with RAG did not reliably follow task-specific instructions, struggled to reason about dates, and could not handle calendar availability or complex multi-step searches.

Workflow diagram · grounded in source
1
User submits query
trigger
“When a user submits a query, two parallel processes occur for the Ask AI agent”
2
Tool classification
ai_action
“Tool classification: The system classifies the query based on user intent to determine which tools or data sources to activate. The classifier identifies whether the query requires: 1) Email search only 2) Email + calendar event search 3…”
3
Metadata extraction
ai_action
“Simultaneously, the system extracts relevant tool parameters such as time filters, sender names, or relevant attachments. These will be used in retrieval to narrow the scope of search to improve accuracy.”
4
Hybrid search and reranking
ai_action
“If the task required search, it would be passed into the search tool (with a hybrid semantic + keyword search) with reranking algorithms to prioritize the most relevant information.”
5
Task-specific response generation
ai_action
“Based on the classification in step 1, the system would select different prompts and preferences. Prompts would contain context-specific instructions with query-specific examples, and also encoded user preferences. The LLM, guided by a s…”
6
Uncertain result validation
validation
“Ask AI would also validate uncertain results with the user before providing a final answer”
Reported outcome

Ask AI cut user search time by 5 minutes per week, a 14% savings, while achieving sub-2-second response times and reduced hallucinations through post-processing.

Reported metrics
weekly email search time (before Ask AI)up to 35 minutes per week
Search time saved per week5 minutes every week
Time savings on search14%
Response time targetSub-2-second responses
Reported stack
RAG
Source
https://www.langchain.com/breakoutagents/superhuman
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Ask AI cut user search time by 5 minutes per week, a 14% savings, while achieving sub-2-second response times and reduced hallucinations through post-processing.

What tools did this team use?

RAG.

What results were reported?

weekly email search time (before Ask AI): up to 35 minutes per week; Search time saved per week: 5 minutes every week; Time savings on search: 14%; Response time target: Sub-2-second responses (source-reported, not independently verified).

What failed first in this deployment?

The initial single-prompt LLM with RAG did not reliably follow task-specific instructions, struggled to reason about dates, and could not handle calendar availability or complex multi-step searches.

How is this back office ops AI workflow structured?

User submits query → Tool classification → Metadata extraction → Hybrid search and reranking → Task-specific response generation → Uncertain result validation.

Related back office ops cases