Back office ops · Production

Two years of vector search at Notion: 10x scale with 90% cost reduction

The problem

Notion AI Q&A launched in November 2023 to overwhelming demand, creating a waitlist of millions of workspaces. The original pod-based vector infrastructure neared storage capacity within one month of launch, and daily onboarding was so slow that clearing the backlog at the initial rate would have taken decades.

First attempt

The original dedicated-hardware pod architecture coupled storage and compute, making over-provisioning prohibitively expensive and requiring complex incremental re-sharding every two weeks. Managing multiple database generations became operationally complex and expensive during the growth phase.

Workflow diagram · grounded in source
1
Dual-path ingestion pipeline
integration
“our ingestion indexing pipeline had two paths: Offline path: Batch jobs running on Apache Spark that chunk existing documents, generate embeddings via API, and bulk-load vectors into our vector database Online path: Real-time updates via…”
2
Generation-based shard routing
routing
“when a set of indexes approached capacity, we'd provision a new set and direct all new workspace onboarding there. Each set got a "generation" ID that determined where reads and writes would go”
3
Serverless architecture migration
integration
“In May 2024, we migrated our entire embeddings workload from the pre-existing dedicated-hardware "pod" architecture to a new serverless architecture that decoupled storage from compute and charged based on usage rather than uptime. The b…”
4
Turbopuffer migration with model upgrade
integration
“we committed to migrating our entire multi billion object workload to turbopuffer in late 2024. Since we were switching providers, we took the opportunity to comprehensively overhaul our overall architecture: Full re-indexing: We increas…”
5
Hash-based span change detection
validation
“we kept track of two hashes per span: one on the span text, and the other on all of the metadata fields. We used the 64-bit variant of the xxHash algorithm because it balanced ease of use, speed, and low collisions with the storage footp…”
6
Selective re-embedding
ai_action
“We detect exactly which spans changed, and only re-embed and re-load those spans into our vector database. We detect that all the text hashes are the same, but all the metadata hashes are different. This means we can skip embedding altog…”
7
Ray/Anyscale embeddings pipeline
integration
“In July 2025, we set out to migrating our near real-time embeddings pipeline to Ray running on Anyscale. Unified compute: By consolidating preprocessing and inference onto a single compute layer, we eliminated the double-compute problem.…”
8
Query-time embedding serving
ai_action
“Ray Serve provides most of this out-of-the-box. It allows us to wrap our open-source embedding models in a persistent deployment that stays loaded on the GPU. We can configure everything from dynamic request batching to replication. The …”
Reported outcome

Over two years, Notion scaled its vector search infrastructure by 10x while reducing costs by 90 percent, achieving a 600x increase in daily onboarding capacity and clearing the Q&A waitlist by April 2024.
p50 query latency improved from 70–100ms to 50–70ms, and hash-based selective re-embedding achieved a 70% reduction in data volume.

Reported metrics
Vector search infrastructure scale10x
Overall cost reduction90 percent
Daily onboarding capacity600x increase
Active workspaces growth15x growth
Show all 12 reported metrics
vector search infrastructure scale10x
overall cost reduction90 percent
daily onboarding capacity600x increase
active workspaces growth15x growth
vector database capacity expansion8x expansion
cost reduction from serverless migration50 percent
annual savings from serverlessseveral millions of dollars saved annually
search engine spend reduction (turbopuffer)60 percent
AWS EMR compute cost reduction35 percent
p50 production query latencyimproved from 70-100ms to 50-70ms
data volume reduction (Page State Project)70%
embeddings infrastructure cost reduction (anticipated)90+ percent
Reported stack
Apache SparkKafkaAirflowturbopufferDynamoDBRayAnyscaleAWS EMRxxHashRay ServeSlackGoogle Drive
Source
https://www.notion.com/blog/two-years-of-vector-search-at-notion
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Over two years, Notion scaled its vector search infrastructure by 10x while reducing costs by 90 percent, achieving a 600x increase in daily onboarding capacity and clearing the Q&A waitlist by April 2024.

What tools did this team use?

Apache Spark, Kafka, Airflow, turbopuffer, DynamoDB, Ray, Anyscale, AWS EMR, xxHash, Ray Serve.

What results were reported?

Vector search infrastructure scale: 10x; Overall cost reduction: 90 percent; Daily onboarding capacity: 600x increase; Active workspaces growth: 15x growth (source-reported, not independently verified).

What failed first in this deployment?

The original dedicated-hardware pod architecture coupled storage and compute, making over-provisioning prohibitively expensive and requiring complex incremental re-sharding every two weeks.

How is this back office ops AI workflow structured?

Dual-path ingestion pipeline → Generation-based shard routing → Serverless architecture migration → Turbopuffer migration with model upgrade → Hash-based span change detection → Selective re-embedding → Ray/Anyscale embeddings pipeline → Query-time embedding serving.

Related back office ops cases