Grammarly builds a compact on-device spelling and grammar correction model (~1B parameters)
Grammarly's writing assistance goes offline when internet is unavailable because its corrections rely on multiple large models that cannot run on a user's device due to limited memory and processing capabilities.
T5, an encoder-decoder model evaluated as a base model candidate, failed the tokenization requirement by converting nonstandard spaces to regular spaces, making it unsuitable.
A compact ~1B parameter model achieves ~210 tokens/second on an M2 Mac with a 70% reduced memory footprint, enabling real-time on-device spelling and grammar corrections without loss in quality.
Frequently asked questions
What did this team achieve with this AI workflow?
A compact ~1B parameter model achieves ~210 tokens/second on an M2 Mac with a 70% reduced memory footprint, enabling real-time on-device spelling and grammar corrections without loss in quality.
What tools did this team use?
Llama, T5, MLX, GQA.
What results were reported?
On-device processing speed: ~210 tokens/second; Memory footprint reduction from quantization: 70%; Performance threshold for real-time correction: 50 tokens per second; Model parameter count: ~1B parameters (source-reported, not independently verified).
What failed first in this deployment?
T5, an encoder-decoder model evaluated as a base model candidate, failed the tokenization requirement by converting nonstandard spaces to regular spaces, making it unsuitable.
How is this back office ops AI workflow structured?
User writes in real time → Llama tokenizes input → Model corrects spelling and grammar → Suggestions delivered inline → Human annotator evaluation → Small-user-set feedback rollout.