Data entry ops · Production

Dropbox uses machine learning and OCR to make text in billions of images searchable

The problem

Images and image-embedded PDFs stored by Dropbox users were invisible to search indexing because they contain only pixels rather than extractable text, leaving billions of files—including receipts, whiteboard photos, and scanned documents—unsearchable.

First attempt

An initial deployed pipeline version was computationally prohibitive—requiring an enormous cluster—and actual traffic was roughly twice the projected load; TensorFlow's default multicore behavior caused severe context-switching overhead that degraded throughput further.

Workflow diagram · grounded in source
1
File event ingestion
trigger
“we need a system that can ingest incoming file events (e.g., adds or edits) and kick off the relevant processing. This turns out to be a natural use case for Cape”
2
Eligibility and type check
validation
“Check whether we should continue processing, based on whether it's a JPEG, GIF, TIFF, or PDF without embedded text and if the user is eligible for the feature”
3
OCR-able classification
ai_action
“The model we trained was a convolutional neural network which takes an input image before converting its output into a binary decision about whether it is likely to have text content”
4
Document corner detection
ai_action
“The document corner detector component was built using another ImageNet deep convolutional network (Densenet-121), with its top layer replaced by a regressor that produces quad corner coordinates”
5
OCR token extraction
ai_action
“It takes rectified images from the corner detection step as input and generates token detections, which include bounding boxes for the tokens and the text of each token”
6
Search index update
output
“Add the list of tokens to the user-specific search index”
Reported outcome

Dropbox deployed automatic image text recognition for Professional and Business Advanced/Enterprise plan users, achieving a throughput improvement of about 3x through TensorFlow tuning and an 88% reduction in PDF metadata extraction failures, with almost 90% of documents indexed completely.

Reported metrics
OCR throughput improvementabout 3x
Densenet-121 speed vs previous modelalmost twice as fast
Documents indexed completelyalmost 90%
image and PDF files stored in Dropboxmore than 20 billion
Show all 6 reported metrics
OCR throughput improvementabout 3x
Densenet-121 speed vs previous modelalmost twice as fast
documents indexed completelyalmost 90%
image and PDF files stored in Dropboxmore than 20 billion
JPEGs likely containing textroughly 9%
actual traffic vs estimatedabout 2x what we estimated
Reported stack
OCRCapePDFiumCaffeTensorFlow XLA
Source
https://dropbox.tech/machine-learning/using-machine-learning-to-index-text-from-billions-of-images
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Dropbox deployed automatic image text recognition for Professional and Business Advanced/Enterprise plan users, achieving a throughput improvement of about 3x through TensorFlow tuning and an 88% reduction in PDF meta…

What tools did this team use?

OCR, Cape, PDFium, Caffe, TensorFlow XLA.

What results were reported?

OCR throughput improvement: about 3x; Densenet-121 speed vs previous model: almost twice as fast; Documents indexed completely: almost 90%; image and PDF files stored in Dropbox: more than 20 billion (source-reported, not independently verified).

What failed first in this deployment?

An initial deployed pipeline version was computationally prohibitive—requiring an enormous cluster—and actual traffic was roughly twice the projected load; TensorFlow's default multicore behavior caused severe context…

How is this data entry ops AI workflow structured?

File event ingestion → Eligibility and type check → OCR-able classification → Document corner detection → OCR token extraction → Search index update.

Related data entry ops cases