Back office ops · Production

Nautilus: Dropbox's ML-powered full-text search engine architecture

The problem

Dropbox needed a new search engine capable of handling its massive-scale document corpus with personalized, near-real-time results tailored to each user's access permissions and behaviors.

Workflow diagram · grounded in source
1
File/user activity trigger
trigger
“As users interact with files and each other, such as editing files or sharing them with other users, we generate "index mutations"”
2
Document content extraction
integration
“we rely on Apache Tika to transform the original document into a canonical HTML representation, which then gets parsed in order to extract a list of "tokens" (i.e. words) and their "attributes" (i.e. formatting, position, etc…)”
3
ML doc understanding pipeline
ai_action
“we can augment the data in various ways using a "Doc Understanding" pipeline, which is well suited for experimenting with extraction of optional metadata and signals. As input it takes the data extracted from the document itself and outp…”
4
Periodic offline index build
output
“We generate "offline" builds of the search index on a regular basis (every 3 days, on average)”
5
Real-time index mutations via Kafka
integration
“is then continuously updated by applying mutations consumed from Kafka queues”
6
Access control scope check
validation
“the first task performed by Octopus is to call Dropbox's access-control service to determine the exact set of namespaces the user has read access to”
7
Distributed document retrieval
integration
“The root is primarily in charge of fanning out incoming queries to the set of leaves holding the data, and then receiving and merging results from the leaves”
8
ML-based result ranking
ai_action
“The ranking engine is powered by a ML model that outputs a score for each document based on a variety of signals”
9
Model retraining feedback loop
feedback_loop
“the model is retrained or updated frequently, adapting and learning from general users' behaviors over time”
Reported outcome

Nautilus became the primary search engine at Dropbox after a shadow-mode qualification period, delivering significant improvements to time-to-index new and updated content.

Reported metrics
Time-to-index improvementsignificant improvements to the time-to-index new and updated content
95th percentile search latency budget500ms
Reported stack
NautilusApache TikaKafkaOctopusBM25
Source
https://dropbox.tech/machine-learning/architecture-of-nautilus-the-new-dropbox-search-engine
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Nautilus became the primary search engine at Dropbox after a shadow-mode qualification period, delivering significant improvements to time-to-index new and updated content.

What tools did this team use?

Nautilus, Apache Tika, Kafka, Octopus, BM25.

What results were reported?

Time-to-index improvement: significant improvements to the time-to-index new and updated content; 95th percentile search latency budget: 500ms (source-reported, not independently verified).

How is this back office ops AI workflow structured?

File/user activity trigger → Document content extraction → ML doc understanding pipeline → Periodic offline index build → Real-time index mutations via Kafka → Access control scope check → Distributed document retrieval → ML-based result ranking → Model retraining feedback loop.

Related back office ops cases