Data entry ops · Production

Dropbox builds in-house deep learning OCR pipeline for mobile document scanner

The problem

Dropbox's commercial off-the-shelf OCR SDK was expensive (charged per scan) and tuned for flat-bed scanners, not mobile photos with crinkled documents, shadows, and uneven lighting; Dropbox also lacked control over future innovation.

First attempt

When the Word Detector and Word Deep Net were first chained end-to-end, accuracy dropped to around 44%—far below the competition—due to spacing errors and spurious garbage text from image noise.

Workflow diagram · grounded in source
1
Mobile document upload
trigger
“mobile clients upload scanned document images to our in-house asynchronous work queue. When the upload is finished, we then send the image via a Remote Procedure Call (RPC) to a cluster of servers running the OCR service”
2
Orientation detection
ai_action
“We built an orientation predictor using another deep net based on the Inception Resnet v2 architecture, changed the final layer to predict orientation, collected an orientation training and validation data set, and fine-tuned from an Ima…”
3
Word Detector segments image
ai_action
“Our Word Detector first detects MSER features in an image, then strings these together into word and line detections”
4
Word Deep Net recognizes text
ai_action
“Convolutional Neural Net (CNN)with several convolutional layers. The visual features that are output by the CNN are then fed as a sequence to a Bidirectional LSTM ( Long Short Term Memory ) — common in speech recognition systems — which …”
5
Confidence scoring and lexicon validation
validation
“If the confidence was high, we kept the prediction as is. If the confidence was low, we simply filtered them out, making a bet that these were noise predictions. If the confidence was somewhere in the middle, we then ran it through a lex…”
6
Wordinator post-processing
output
“we run these outputs along with the original outputs from the Word Detector through a module we call the Wordinator, which gives discrete bounding boxes for each individual OCRed word. This results in individual word coordinates along wi…”
7
PDF hidden layer and search index
output
“we merge them back into the original PDF produced by the mobile document scanner as an OCR hidden layer. The OCRed text is also added to Dropbox's search index. The user can now highlight and copy-paste text from the PDF, with the highli…”
Reported outcome

After about 8 months of research, productionization, and refinement, Dropbox deployed a state-of-the-art OCR pipeline to millions of Dropbox Business users, achieving mid-90s Single Word Accuracy and replacing the commercial SDK entirely.

Reported metrics
Single Word Accuracy after initial synthetic trainingaround 79%
Single Word Accuracy after fine-tuning on real imagesmid-90s
Initial end-to-end accuracy44%
deployment coverage of Dropbox Business users100%
Show all 6 reported metrics
Single Word Accuracy after initial synthetic trainingaround 79%
Single Word Accuracy after fine-tuning on real imagesmid-90s
initial end-to-end accuracy44%
deployment coverage of Dropbox Business users100%
cost vs commercial SDKsignificant money
total development timeabout 8 months
Reported stack
TensorFlowOpenCVTorchAmazon EC2 G2S3StormcrowDropTurkMechanical TurkLXCInception Resnet v2
Source
https://dropbox.tech/machine-learning/creating-a-modern-ocr-pipeline-using-computer-vision-and-deep-learning
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After about 8 months of research, productionization, and refinement, Dropbox deployed a state-of-the-art OCR pipeline to millions of Dropbox Business users, achieving mid-90s Single Word Accuracy and replacing the com…

What tools did this team use?

TensorFlow, OpenCV, Torch, Amazon EC2 G2, S3, Stormcrow, DropTurk, Mechanical Turk, LXC, Inception Resnet v2.

What results were reported?

Single Word Accuracy after initial synthetic training: around 79%; Single Word Accuracy after fine-tuning on real images: mid-90s; Initial end-to-end accuracy: 44%; deployment coverage of Dropbox Business users: 100% (source-reported, not independently verified).

What failed first in this deployment?

When the Word Detector and Word Deep Net were first chained end-to-end, accuracy dropped to around 44%—far below the competition—due to spacing errors and spurious garbage text from image noise.

How is this data entry ops AI workflow structured?

Mobile document upload → Orientation detection → Word Detector segments image → Word Deep Net recognizes text → Confidence scoring and lexicon validation → Wordinator post-processing → PDF hidden layer and search index.

Related data entry ops cases