Back office ops · Production

Two years of vector search at Notion: 10x scale with 90% cost reduction

The problem

Notion AI Q&A launched in November 2023 to overwhelming demand, creating a waitlist of millions of workspaces. The original pod-based vector infrastructure neared storage capacity within one month of launch, and daily onboarding was so slow that clearing the backlog at the initial rate would have taken decades.

First attempt

The original dedicated-hardware pod architecture coupled storage and compute, making over-provisioning prohibitively expensive and requiring complex incremental re-sharding every two weeks. Managing multiple database generations became operationally complex and expensive during the growth phase.

Workflow diagram · grounded in source
1
Dual-path ingestion pipeline
Integration
The indexing pipeline runs two paths: offline Spark batch jobs that chunk documents and bulk-load vectors, and online Kafka consumers that process individual page edits in real time.
▾ source quote
“our ingestion indexing pipeline had two paths: Offline path: Batch jobs running on Apache Spark that chunk existing documents, generate embeddings via API, and bulk-load vectors into our vector database Online path: Real-time updates via Kafka consumers that process individual …”
2
Generation-based shard routing
Routing
When a set of indexes approaches capacity, a new set is provisioned and assigned a generation ID that determines where reads and writes are directed.
▾ source quote
“when a set of indexes approached capacity, we'd provision a new set and direct all new workspace onboarding there. Each set got a "generation" ID that determined where reads and writes would go”
3
Serverless architecture migration
Integration
In May 2024, the embeddings workload migrated from dedicated-hardware pod clusters to a serverless architecture that decouples storage from compute and charges based on usage rather than uptime.
▾ source quote
“In May 2024, we migrated our entire embeddings workload from the pre-existing dedicated-hardware "pod" architecture to a new serverless architecture that decoupled storage from compute and charged based on usage rather than uptime. The benefits were immediate and substantial: A …”
4
turbopuffer migration with model upgrade
Integration
The entire multi-billion object workload was migrated to turbopuffer with full re-indexing, an embeddings model upgrade, and a gradual cutover that validated correctness one generation at a time.
▾ source quote
“we committed to migrating our entire multi billion object workload to turbopuffer in late 2024. Since we were switching providers, we took the opportunity to comprehensively overhaul our overall architecture: Full re-indexing: We increased our writes throughput in our offline …”
5
Hash-based span change detection
Validation
Two hashes per span—one on text and one on metadata—are stored in DynamoDB using xxHash to detect exactly which spans changed and what re-processing is required.
▾ source quote
“we kept track of two hashes per span: one on the span text, and the other on all of the metadata fields. We used the 64-bit variant of the xxHash algorithm because it balanced ease of use, speed, and low …”
6
Selective re-embedding
Ai action
Only spans whose text hash changed are re-embedded and reloaded; spans with only changed metadata receive a cheaper PATCH command, skipping embedding entirely.
▾ source quote
“We detect exactly which spans changed, and only re-embed and re-load those spans into our vector database. We detect that all the text hashes are the same, but all the metadata hashes are different. This means we can skip embedding …”
7
Ray/Anyscale embeddings pipeline
Integration
The near real-time embeddings pipeline was migrated to Ray on Anyscale, consolidating preprocessing and inference onto a single compute layer and enabling direct hosting of open-source embedding models.
▾ source quote
“In July 2025, we set out to migrating our near real-time embeddings pipeline to Ray running on Anyscale. Unified compute: By consolidating preprocessing and inference onto a single compute layer, we eliminated the double-compute problem. Model flexibility: Ray lets us …”
8
Query-time embedding serving
Ai action
Ray Serve wraps open-source embedding models in a persistent GPU-loaded deployment supporting dynamic request batching, replication, and autoscaling for low-latency query-time embedding.
▾ source quote
“Ray Serve provides most of this out-of-the-box. It allows us to wrap our open-source embedding models in a persistent deployment that stays loaded on the GPU. We can configure everything from dynamic request batching to replication. The model serving code …”
Reported outcome

Over two years, Notion scaled its vector search infrastructure by 10x while reducing costs by 90 percent, achieving a 600x increase in daily onboarding capacity and clearing the Q&A waitlist by April 2024.
p50 query latency improved from 70–100ms to 50–70ms, and hash-based selective re-embedding achieved a 70% reduction in data volume.

Reported metrics
Vector search infrastructure scale10x
Overall cost reduction90 percent
Daily onboarding capacity600x increase
Active workspaces growth15x growth
Show all 12 reported metrics
vector search infrastructure scale10x
overall cost reduction90 percent
daily onboarding capacity600x increase
active workspaces growth15x growth
vector database capacity expansion8x expansion
cost reduction from serverless migration50 percent
annual savings from serverlessseveral millions of dollars saved annually
search engine spend reduction (turbopuffer)60 percent
AWS EMR compute cost reduction35 percent
p50 production query latencyimproved from 70-100ms to 50-70ms
data volume reduction (Page State Project)70%
embeddings infrastructure cost reduction (anticipated)90+ percent
Reported stack
Apache SparkKafkaAirflowturbopufferDynamoDBRayAnyscaleAWS EMRxxHashRay ServeSlackGoogle Drive
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://www.notion.com/blog/two-years-of-vector-search-at-notion
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Over two years, Notion scaled its vector search infrastructure by 10x while reducing costs by 90 percent, achieving a 600x increase in daily onboarding capacity and clearing the Q&A waitlist by April 2024.

What tools did this team use?

Apache Spark, Kafka, Airflow, turbopuffer, DynamoDB, Ray, Anyscale, AWS EMR, xxHash, Ray Serve.

What results were reported?

Vector search infrastructure scale: 10x; Overall cost reduction: 90 percent; Daily onboarding capacity: 600x increase; Active workspaces growth: 15x growth (source-reported, not independently verified).

What failed first in this deployment?

The original dedicated-hardware pod architecture coupled storage and compute, making over-provisioning prohibitively expensive and requiring complex incremental re-sharding every two weeks.

How is this back office ops AI workflow structured?

Dual-path ingestion pipeline → Generation-based shard routing → Serverless architecture migration → Turbopuffer migration with model upgrade → Hash-based span change detection → Selective re-embedding → Ray/Anyscale embeddings pipeline → Query-time embedding serving.

Related back office ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Back office ops cases →