Data entry ops · Production

Dropbox improves document scanner responsiveness with hybrid quad tracking

The problem

Dropbox's document detection algorithm required 100 ms per frame while the camera captures at 30 fps, making real-time quad overlay impossible — especially on older devices like iPhone 5 that lacked the processing power of newer hardware.

First attempt

Asynchronous processing showed quads offset 100 ms from the displayed image, causing laggy and choppy overlays. Synchronous processing reduced frame rate to 10 fps with jarring 100 ms latency. Several tracking approaches — brute-forcing, RANSAC, and digest-based alignment — were not fast enough.

Workflow diagram · grounded in source
1
Camera frame capture
trigger
“the camera pumps out images at 30 frames per second (fps) continuously”
2
Asynchronous quad detection
ai_action
“we continue to run the quad detector as before, in a loop—it will now take slightly over 100 ms, given the extra compute we are performing—to provide the latest accurate estimate of the quad”
3
Camera motion estimation
ai_action
“We rely on the onboard gyroscope to compute the roll of the camera between consecutive frames, which can then be factored out, so the problem is reduced to that of finding a scaled and translated version of a particular quadrilateral”
4
Quad tracking via gradient optimization
ai_action
“we try to find the transform of this quad such that the edge response of the hypothetical new quad, defined to be the line integral of the gradient of the image measured perpendicular to the perimeter of the quad, is maximized”
5
Synced display with quad overlay
output
“The image is held until this (quick) tracking process is done, and is displayed along with the quad on the screen”
Reported outcome

The hybrid approach achieves 30 Hz image and quad throughput with approximately 30 ms latency and zero image-to-quad offset, combining the frame-rate of asynchronous mode with the quad accuracy of synchronous mode.

Reported metrics
Detection algorithm time per frame100ms
Camera frame rate30 fps
Synchronous image throughput10 Hz
Synchronous image latency100 ms
Show all 10 reported metrics
detection algorithm time per frame100ms
camera frame rate30 fps
synchronous image throughput10 Hz
synchronous image latency100 ms
asynchronous image vs quad offset100 ms
hybrid image throughput30 Hz
hybrid image latency~30 ms
hybrid quad throughput30 Hz
hybrid quad latency~30 ms
hybrid image vs quad offset0 ms
Reported stack
gyroscope
Source
https://dropbox.tech/machine-learning/improving-the-responsiveness-of-the-document-detector
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The hybrid approach achieves 30 Hz image and quad throughput with approximately 30 ms latency and zero image-to-quad offset, combining the frame-rate of asynchronous mode with the quad accuracy of synchronous mode.

What tools did this team use?

gyroscope.

What results were reported?

Detection algorithm time per frame: 100ms; Camera frame rate: 30 fps; Synchronous image throughput: 10 Hz; Synchronous image latency: 100 ms (source-reported, not independently verified).

What failed first in this deployment?

Asynchronous processing showed quads offset 100 ms from the displayed image, causing laggy and choppy overlays.

How is this data entry ops AI workflow structured?

Camera frame capture → Asynchronous quad detection → Camera motion estimation → Quad tracking via gradient optimization → Synced display with quad overlay.

Related data entry ops cases