Workflow · Production

Half-Quadratic Quantization (HQQ): Calibration-Free Quantization of Large ML Models

The problem

Deploying large language models is memory-intensive, and while calibration-based quantization methods like GPTQ and AWQ offer better quality than data-free approaches, they suffer from calibration data bias and prohibitively slow processing times on the largest models.

First attempt

Gradient-based calibration-free optimization with autograd requires many iterations and fails when using the sparsity-promoting norms (p<1) needed to handle weight outliers effectively.

Workflow diagram · grounded in source
1
Sparsity-promoting objective
ai_action
“our method focuses specifically on minimizing errors in the weights rather than the layer activation. Additionally, by incorporating a sparsity-promoting loss, such as the \( {l_{p<1}} \)-norm, we effectively model outliers through a hyp…”
2
Half-Quadratic problem split
ai_action
“we adopt a Half-Quadratic solver by introducing an extra variable W_e. This additional parameter allows us to split the main problem into sub-problems that are easier to solve”
3
Alternate optimization
ai_action
“the approach that we propose relies on closed-form solutions, which means that there are no gradients calculated. This allows us to run all the calculations in inference mode with half-precision. Moreover, it only takes a few iterations …”
4
Perplexity and accuracy benchmark
validation
“we use the perplexity metric (PPL) on the widely adopted wikitext2 dataset. We also report the runtime GPU memory in GB (MEM) the session takes to run the quantized model”
5
Open-source model release
output
“Ready-to-use quantized models can be found on our Hugging Face 🤗 page: https://huggingface.co/mobiuslabsgmbh”
Reported outcome

HQQ achieves calibration-free quantization quality competitive with GPTQ and AWQ, quantizing Llama-2-70B in under 5 minutes — over 50x faster than GPTQ — with 2-bit HQQ Llama-2-70B outperforming full-precision Llama-2-13B at comparable memory usage.

Reported metrics
Llama-2-70B quantization timeless than 5 minutes
HQQ speed vs GPTQ for Llama-2-70Bover 50x faster
HQQ speed vs autograd for Llama-2-7Bover 100x faster
2-bit HQQ Llama-2-70B vs full-precision Llama-2-13B perplexityoutperforms the full-precision Llama-2-13B by a large margin
Show all 8 reported metrics
Llama-2-70B quantization timeless than 5 minutes
HQQ speed vs GPTQ for Llama-2-70Bover 50x faster
HQQ speed vs autograd for Llama-2-7Bover 100x faster
2-bit HQQ Llama-2-70B vs full-precision Llama-2-13B perplexityoutperforms the full-precision Llama-2-13B by a large margin
4-bit ViT-B-32 zero-shot top-1 accuracy vs BNB 4-bit+3.1%
3-bit ViT-H-14 vs full-precision ViT-L-14 zero-shot top-1+2.4%
2-bit ViT-H-14 vs full-precision ViT-B-32 zero-shot top-1+5.2%
autograd iterations requiredthousands of iterations
Reported stack
bitsandbytesGPTQAWQLlama-2OpenCLIPAutoGPTQAutoAWQ
Source
https://dropbox.tech/machine-learning/halfquadratic-quantization-of-large-machine-learning-models
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

HQQ achieves calibration-free quantization quality competitive with GPTQ and AWQ, quantizing Llama-2-70B in under 5 minutes — over 50x faster than GPTQ — with 2-bit HQQ Llama-2-70B outperforming full-precision Llama-2…

What tools did this team use?

bitsandbytes, GPTQ, AWQ, Llama-2, OpenCLIP, AutoGPTQ, AutoAWQ.

What results were reported?

Llama-2-70B quantization time: less than 5 minutes; HQQ speed vs GPTQ for Llama-2-70B: over 50x faster; HQQ speed vs autograd for Llama-2-7B: over 100x faster; 2-bit HQQ Llama-2-70B vs full-precision Llama-2-13B perplexity: outperforms the full-precision Llama-2-13B by a large margin (source-reported, not independently verified).

What failed first in this deployment?

Gradient-based calibration-free optimization with autograd requires many iterations and fails when using the sparsity-promoting norms (p<1) needed to handle weight outliers effectively.

How is this workflow AI workflow structured?

Sparsity-promoting objective → Half-Quadratic problem split → Alternate optimization → Perplexity and accuracy benchmark → Open-source model release.