Workflow · Production

Grammarly builds CoEdIT: an instruction-tuned LLM for text editing that outperforms GPT-3-Edit while being up to 60 times smaller

The problem

General-purpose LLMs were trained for broad text-generation tasks and lacked instruction tuning for text editing, limiting their usability, quality, and performance on well-scoped editing tasks.

First attempt

Prior text editing LLMs suffered from four identified gaps: no instruction tuning, undersized models, highly general (not task-specific) training datasets, and lack of public availability.

Workflow diagram · grounded in source
1
Identify LLM research gaps
trigger
“gaps present in developing general-purpose text editing models using LLMs, as they significantly limited model effectiveness, performance, or usability”
2
Build CoEdIT training dataset
output
“we built upon the IteraTeR+ dataset1, which contains a variety of text editing tasks while focusing on non-meaning-changing edits. We translated edit categories—Fluency, Coherence, Clarity, Style—into natural language instructions, such …”
3
Fine-tune FLAN-T5 models
ai_action
“we fine-tuned a few different versions of a pre-trained FLANT5 LLM (L: 770 million parameters, XL: 3 billion parameters, XXL: 11 billion parameters) with the CoEdIT dataset. We named these models CoEdIT-L, CoEdIT-XL, and CoEdIT-XXL respe…”
4
Quantitative benchmark evaluation
validation
“Representatives from each of these four comparison groups, along with CoEdIT, were then evaluated against standard test sets from a variety of text editing benchmarks”
5
Human expert evaluation
human_review
“Our expert evaluators compared the outputs of two models, CoEdIT-XL (3 billion) and GPT3-Edit (175 billion), for fluency, accuracy, and preservation of meaning”
6
Composite task training and evaluation
validation
“we enriched its training set with multi-part tasks, like "grammatical-error-correction with paraphrasing and simplification." This led to the development of CoEdIT-Composite, trained on this set of composite tasks.”
Reported outcome

CoEdIT achieves state-of-the-art performance on multiple benchmark test sets while being up to 60 times smaller than comparable LLMs.
Human evaluators preferred CoEdIT's output 64 percent of the time compared to 10 percent for GPT3-Edit.

Reported metrics
model size vs comparable LLMsup to 60 times smaller
Parameter count vs comparable performers12 times and 60 times fewer parameters
human evaluator preference for CoEdIT64 percent
human evaluator preference for GPT3-Edit10 percent
Show all 7 reported metrics
model size vs comparable LLMsup to 60 times smaller
parameter count vs comparable performers12 times and 60 times fewer parameters
human evaluator preference for CoEdIT64 percent
human evaluator preference for GPT3-Edit10 percent
CoEdIT-Composite preference vs GPT3-Edit38 percent to 34 percent
CoEdIT-Composite preference vs CoEdIT-XL34 percent to 21 percent
benchmark performance ratingstate-of-the-art performance on multiple benchmark test sets
Reported stack
CoEdITGPT3-EditChatGPT
Source
https://www.grammarly.com/blog/engineering/coedit-text-editing/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

CoEdIT achieves state-of-the-art performance on multiple benchmark test sets while being up to 60 times smaller than comparable LLMs.

What tools did this team use?

CoEdIT, GPT3-Edit, ChatGPT.

What results were reported?

model size vs comparable LLMs: up to 60 times smaller; Parameter count vs comparable performers: 12 times and 60 times fewer parameters; human evaluator preference for CoEdIT: 64 percent; human evaluator preference for GPT3-Edit: 10 percent (source-reported, not independently verified).

What failed first in this deployment?

Prior text editing LLMs suffered from four identified gaps: no instruction tuning, undersized models, highly general (not task-specific) training datasets, and lack of public availability.

How is this workflow AI workflow structured?

Identify LLM research gaps → Build CoEdIT training dataset → Fine-tune FLAN-T5 models → Quantitative benchmark evaluation → Human expert evaluation → Composite task training and evaluation.