Data entry ops · Production

Dropbox improves document scanner responsiveness with hybrid quad tracking

The problem

Dropbox's document detection algorithm required 100 ms per frame while the camera captures at 30 fps, making real-time quad overlay impossible — especially on older devices like iPhone 5 that lacked the processing power of newer hardware.

First attempt

Asynchronous processing showed quads offset 100 ms from the displayed image, causing laggy and choppy overlays. Synchronous processing reduced frame rate to 10 fps with jarring 100 ms latency. Several tracking approaches — brute-forcing, RANSAC, and digest-based alignment — were not fast enough.

Workflow diagram · grounded in source
1
Camera frame capture
Trigger
The camera continuously pumps out images at 30 fps, triggering the detection pipeline.
▾ source quote
“the camera pumps out images at 30 frames per second (fps) continuously”
2
Asynchronous quad detection
Ai action
The document detection algorithm runs continuously in a loop to provide the latest accurate estimate of the document quad.
▾ source quote
“we continue to run the quad detector as before, in a loop—it will now take slightly over 100 ms, given the extra compute we are performing—to provide the latest accurate estimate of the quad”
3
Camera motion estimation
Ai action
The onboard gyroscope computes the roll between consecutive frames, which is factored out to reduce the motion problem to scale and translation.
▾ source quote
“We rely on the onboard gyroscope to compute the roll of the camera between consecutive frames, which can then be factored out, so the problem is reduced to that of finding a scaled and translated version of a particular quadrilateral”
4
Quad tracking via gradient optimization
Ai action
The previous frame's quad is transformed to find its position in the current frame by maximizing the line integral of the image gradient along the quad's perimeter.
▾ source quote
“we try to find the transform of this quad such that the edge response of the hypothetical new quad, defined to be the line integral of the gradient of the image measured perpendicular to the perimeter of the quad, is …”
5
Synced display with quad overlay
Output
The image is held until the quick tracking process completes, then displayed with the synced quad overlay.
▾ source quote
“The image is held until this (quick) tracking process is done, and is displayed along with the quad on the screen”
Reported outcome

The hybrid approach achieves 30 Hz image and quad throughput with approximately 30 ms latency and zero image-to-quad offset, combining the frame-rate of asynchronous mode with the quad accuracy of synchronous mode.

Reported metrics
Detection algorithm time per frame100ms
Camera frame rate30 fps
Synchronous image throughput10 Hz
Synchronous image latency100 ms
Show all 10 reported metrics
detection algorithm time per frame100ms
camera frame rate30 fps
synchronous image throughput10 Hz
synchronous image latency100 ms
asynchronous image vs quad offset100 ms
hybrid image throughput30 Hz
hybrid image latency~30 ms
hybrid quad throughput30 Hz
hybrid quad latency~30 ms
hybrid image vs quad offset0 ms
Reported stack
gyroscope
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://dropbox.tech/machine-learning/improving-the-responsiveness-of-the-document-detector
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The hybrid approach achieves 30 Hz image and quad throughput with approximately 30 ms latency and zero image-to-quad offset, combining the frame-rate of asynchronous mode with the quad accuracy of synchronous mode.

What tools did this team use?

gyroscope.

What results were reported?

Detection algorithm time per frame: 100ms; Camera frame rate: 30 fps; Synchronous image throughput: 10 Hz; Synchronous image latency: 100 ms (source-reported, not independently verified).

What failed first in this deployment?

Asynchronous processing showed quads offset 100 ms from the displayed image, causing laggy and choppy overlays.

How is this data entry ops AI workflow structured?

Camera frame capture → Asynchronous quad detection → Camera motion estimation → Quad tracking via gradient optimization → Synced display with quad overlay.

Related data entry ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Data entry ops cases →