Data entry ops · Production

Dropbox implements augmented camera preview for Android document scanner using ML-based edge detection

The problem

Implementing real-time document edge detection on the fragmented Android hardware landscape required processing each camera frame within 80ms — a constraint that the standard NV21-to-RGBA conversion method, taking 300-500ms per frame, could not meet.

First attempt

Converting NV21 camera frames to RGBA bitmaps using the standard Java method took 300-500ms per 1920x1080 frame, making it unacceptable for a pipeline requiring under 80ms per cycle.

Workflow diagram · grounded in source
1
Camera frame arrives
trigger
“Every time a new frame is available (for most devices, it occurs 20-30 times per second), the callback is invoked.”
2
Frame conversion to RGBA
integration
“By applying these two, we were able to get the processing time down to 10-25 ms thanks to the hardware acceleration.”
3
ML document detection
ai_action
“our document detector requires passing a so called rotation matrix. Such matrix essentially provides information about phone movement direction (like tilting), which expedites calculating the next position of the quadrilateral. Knowing t…”
4
Quad coordinates returned
output
“Once the call to document detector returns, we receive coordinates of the four points representing the quadrilateral that delimits the document edges.”
5
Augmented overlay rendered
output
“drawing a blue quadrilateral that symbolizes the document's boundaries in the live preview”
Reported outcome

Using RenderScript intrinsics reduced frame conversion to 10-25ms; the TextureView approach achieved 5-15ms for the 200x200 input; the complete system achieves at least 15 FPS on most Android devices.

Reported metrics
Frame conversion time (standard method)300-500 ms
frame conversion time (RenderScript)10-25 ms
TextureView getBitmap processing time5-15ms
Document detector call time20-100ms
Show all 6 reported metrics
frame conversion time (standard method)300-500 ms
frame conversion time (RenderScript)10-25 ms
TextureView getBitmap processing time5-15ms
document detector call time20-100ms
minimum achieved frame rateat least 15 FPS
per-frame processing budgetno more than 80 ms
Reported stack
RenderScriptScriptIntrinsicResizeScriptIntrinsicYuvtoRGBJNITextureView
Source
https://dropbox.tech/machine-learning/augmented-camera-previews-for-the-dropbox-android-document-scanner
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Using RenderScript intrinsics reduced frame conversion to 10-25ms; the TextureView approach achieved 5-15ms for the 200x200 input; the complete system achieves at least 15 FPS on most Android devices.

What tools did this team use?

RenderScript, ScriptIntrinsicResize, ScriptIntrinsicYuvtoRGB, JNI, TextureView.

What results were reported?

Frame conversion time (standard method): 300-500 ms; frame conversion time (RenderScript): 10-25 ms; TextureView getBitmap processing time: 5-15ms; Document detector call time: 20-100ms (source-reported, not independently verified).

What failed first in this deployment?

Converting NV21 camera frames to RGBA bitmaps using the standard Java method took 300-500ms per 1920x1080 frame, making it unacceptable for a pipeline requiring under 80ms per cycle.

How is this data entry ops AI workflow structured?

Camera frame arrives → Frame conversion to RGBA → ML document detection → Quad coordinates returned → Augmented overlay rendered.

Related data entry ops cases