Back office ops · Production

Validating performance and reliability of the Dropbox Nautilus search engine

The problem

Dropbox's traffic is write-dominated—writes occur 10x more frequently than reads—requiring a search index format optimized for high-frequency mutations while still delivering low-latency query responses for millions of users.

Workflow diagram · grounded in source

File change triggers index write

trigger

“handling both updating the index when files get created, edited, and deleted (those are "writes")”

ML search ranking and content understanding

ai_action

“machine intelligence to scale our search ranking and content understanding models”

Namespace-partitioned prefix retrieval

integration

“we can efficiently run a prefix search for <namespace ID><token>| in the index to get a list of all matching documents”

ACL check and metadata decoration

validation

“fetching metadata from external systems—backed by relational databases—for checking ACLs and "decorating" the search results before returning them. This includes things like listing the folder path, creator, last modified time, etc.”

Results merging and ranking output

output

“Octopus, our results merging and ranking system”

Reported outcome

Nautilus achieves target query latency of under 500ms at the 95th percentile and under 1 second at the 99th percentile, with 2X replication and automatic partition recovery ensuring full availability during failures and maintenance.

Reported metrics

Write-to-read volume ratio10x higher than reads

Index size overhead (exploded vs conventional)~15% larger

Query latency target at 95th percentile500ms

Query latency target at 99th percentile1sec

Show all 6 reported metrics

write-to-read volume ratio10x higher than reads

index size overhead (exploded vs conventional)~15% larger

query latency target at 95th percentile500ms

query latency target at 99th percentile1sec

replication factor2X

hardware over-provisioning buffer15% extra

Reported stack

RocksDBKafkaOctopus

Source

https://dropbox.tech/machine-learning/validating-performance-and-reliability-of-the-new-dropbox-search-engine

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Nautilus achieves target query latency of under 500ms at the 95th percentile and under 1 second at the 99th percentile, with 2X replication and automatic partition recovery ensuring full availability during failures a…

What tools did this team use?

RocksDB, Kafka, Octopus.

What results were reported?

Write-to-read volume ratio: 10x higher than reads; Index size overhead (exploded vs conventional): ~15% larger; Query latency target at 95th percentile: 500ms; Query latency target at 99th percentile: 1sec (source-reported, not independently verified).

How is this back office ops AI workflow structured?

File change triggers index write → ML search ranking and content understanding → Namespace-partitioned prefix retrieval → ACL check and metadata decoration → Results merging and ranking output.