Workflow · Production

Dropbox: low-bit inference enables efficient AI model serving for Dash

The problem

Running large AI models in production for Dropbox Dash requires substantial and growing memory, compute, and energy, and efficiently serving them within practical hardware, cost, and latency constraints is a central engineering challenge.

Workflow diagram · grounded in source
1
Scale-driven efficiency demand
trigger
“as these models continue to grow in size and capability, so does the demand for memory, computing power, and energy”
2
Quantization applied to weights and activations
ai_action
“By quantizing tensors, for example, from 16-bit to 8-bit or 4-bit, the memory footprint is reduced because each element requires fewer bits”
3
Calibration and grouping
validation
“Popular methods such as AWQ and HQQ rely on linear quantization with grouping, a design that balances efficiency with accuracy”
4
GPU hardware matrix core execution
integration
“NVIDIA GPUs use Tensor Cores, while AMD GPUs use Matrix Cores. These dedicated processors are accessed through matrix multiply-accumulate (MMA) instructions”
5
Production AI feature serving in Dash
output
“Dash relies on large-scale models for experiences such as conversational AI, multimodal search, document understanding, and speech processing”
Reported outcome

Dropbox already employs a range of quantization strategies to optimize model deployment and fully utilize modern accelerators for Dash's AI features, though FP4 adoption and framework support remain incomplete across the industry.

Reported metrics
Inference throughput from halved precisionroughly double throughput
Model serving speed and costfaster and cheaper to run
energy efficiency with FP4 (Blackwell vs H100)significant energy savings
Reported stack
Dropbox DashAWQHQQFlash Attention 3Sage AttentionTriton
Source
https://dropbox.tech/machine-learning/how-low-bit-inference-enables-efficient-ai
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Dropbox already employs a range of quantization strategies to optimize model deployment and fully utilize modern accelerators for Dash's AI features, though FP4 adoption and framework support remain incomplete across…

What tools did this team use?

Dropbox Dash, AWQ, HQQ, Flash Attention 3, Sage Attention, Triton.

What results were reported?

Inference throughput from halved precision: roughly double throughput; Model serving speed and cost: faster and cheaper to run; energy efficiency with FP4 (Blackwell vs H100): significant energy savings (source-reported, not independently verified).

How is this workflow AI workflow structured?

Scale-driven efficiency demand → Quantization applied to weights and activations → Calibration and grouping → GPU hardware matrix core execution → Production AI feature serving in Dash.