Back office ops · Production

How Dropbox built the feature store powering real-time ML ranking in Dash

The problem

Dropbox Dash needed a feature store for real-time ML ranking across vast numbers of work documents, but their infrastructure split across on-premises and Spark-native cloud environments ruled out off-the-shelf solutions. The system had to handle massive parallel feature lookups while meeting strict sub-100ms latency budgets and near-real-time freshness requirements.

First attempt

The initial Python-based feature serving layer, built on the Feast SDK, hit CPU-bound JSON parsing bottlenecks and Python's Global Interpreter Lock under high concurrency; switching to multiple processes temporarily improved latency but introduced coordination overhead that capped scalability.

Workflow diagram · grounded in source

User query triggers ranking

trigger

“A single user query doesn't just pull up one document. Instead, it triggers our ranker to evaluate many files, each requiring dozens of behavioral and contextual features.”

Fan-out feature lookups

ai_action

“What starts as one search quickly fans out into thousands of feature lookups across interaction history, metadata, collaboration patterns, and real-time signals.”

Dynovault feature retrieval

integration

“Dynovault handled the instant feature lookups needed for each search query. Co-located with inference workloads and leveraging Dropbox's hybrid cloud infrastructure, Dynovault avoids the delay of public internet calls and reliably delive…”

Go service feature serving

output

“the Go service delivers true concurrency without the coordination costs we hit in Python. Today, it handles thousands of requests per second while adding only ~5–10ms of processing overhead on top of Dynovault's client latency, consisten…”

Real-time signal ingestion

feedback_loop

“If a user opens a document or joins a Slack channel, that signal should show up in their next search—within a few seconds”

Reported outcome

The rewritten Go serving layer handles thousands of requests per second with p95 latencies in the ~25–35ms range; intelligent change detection cut batch update times from more than an hour to under five minutes and reduced write volumes from hundreds of millions to under one million records per run.

Reported metrics

Feature serving latency budgetsub-100ms

Dynovault client-side latency~20ms

Go service processing overhead~5–10ms

P95 feature serving latency~25–35ms

Show all 7 reported metrics

feature serving latency budgetsub-100ms

Dynovault client-side latency~20ms

Go service processing overhead~5–10ms

p95 feature serving latency~25–35ms

batch update timefrom more than an hour to under five minutes

write volumes per batch runfrom hundreds of millions to under one million records per run

feature value change rate per 15-minute window1–5%

Reported stack

FeastDynovaultGoSparkPySparkAWS DynamoDB

Source

https://dropbox.tech/machine-learning/feature-store-powering-realtime-ai-in-dropbox-dash

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The rewritten Go serving layer handles thousands of requests per second with p95 latencies in the ~25–35ms range; intelligent change detection cut batch update times from more than an hour to under five minutes and re…

What tools did this team use?

Feast, Dynovault, Go, Spark, PySpark, AWS DynamoDB.

What results were reported?

Feature serving latency budget: sub-100ms; Dynovault client-side latency: ~20ms; Go service processing overhead: ~5–10ms; P95 feature serving latency: ~25–35ms (source-reported, not independently verified).

What failed first in this deployment?

How is this back office ops AI workflow structured?

User query triggers ranking → Fan-out feature lookups → Dynovault feature retrieval → Go service feature serving → Real-time signal ingestion.