Compliance monitoring · Production

Grammarly develops DeTexD benchmark dataset and RoBERTa-based classifier for delicate text detection

The problem

Existing toxic text detection methods leave a gap for the broader category of delicate text — emotionally charged or potentially triggering writing that may not be explicitly offensive but still carries risk for users and LLMs exposed to it.

First attempt

All evaluated toxic and hate-speech detection methods underperform on delicate text, missing coverage on medical and mental health topics or showing lower precision on texts containing offensive keywords that are not actually delicate.

Workflow diagram · grounded in source

Delicate text data sourcing

integration

“Domain specification: We specifically targeted news websites, forums discussing sensitive topics, and generally controversial forums. Keyword matching: We developed a dictionary of delicate keywords along with a severity rating for each …”

Expert linguist annotation

human_review

“All annotators were expert linguists with previous experience in similar tasks; the final label was decided by majority vote”

Classifier fine-tuning on DeTexD

ai_action

“We began by fine-tuning a classification model on the DeTexD Training dataset. As our base model, we used the RoBERTa-based classifier”

Benchmark evaluation of methods

validation

“We evaluated several methods on precision, recall, and F1 score (a metric that combines precision and recall)”

Public artifact release

output

“We've made our annotation guidelines, annotated benchmark dataset, and baseline model publicly available”

Reported outcome

Grammarly created the DeTexD dataset (40,000 training samples and 1,023 benchmark paragraphs) and a RoBERTa-based baseline model achieving 79.3% F1, outperforming all evaluated methods.
Annotation guidelines, dataset, and baseline model were made publicly available.

Reported metrics

DeTexD training dataset size40,000 samples

DeTexD benchmark dataset size1,023 paragraphs

Baseline model F1 score79.3%

Baseline model precision81.4%

Show all 5 reported metrics

DeTexD training dataset size40,000 samples

DeTexD benchmark dataset size1,023 paragraphs

Baseline model F1 score79.3%

Baseline model precision81.4%

Baseline model recall78.3%

Reported stack

RoBERTaHateBERTGoogle's Perspective APIOpenAPI content filterOpenAPI moderation API

Source

https://www.grammarly.com/blog/engineering/detecting-delicate-text/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Grammarly created the DeTexD dataset (40,000 training samples and 1,023 benchmark paragraphs) and a RoBERTa-based baseline model achieving 79.3% F1, outperforming all evaluated methods.

What tools did this team use?

RoBERTa, HateBERT, Google's Perspective API, OpenAPI content filter, OpenAPI moderation API.

What results were reported?

DeTexD training dataset size: 40,000 samples; DeTexD benchmark dataset size: 1,023 paragraphs; Baseline model F1 score: 79.3%; Baseline model precision: 81.4% (source-reported, not independently verified).

What failed first in this deployment?

How is this compliance monitoring AI workflow structured?

Delicate text data sourcing → Expert linguist annotation → Classifier fine-tuning on DeTexD → Benchmark evaluation of methods → Public artifact release.

Grammarly created the DeTexD dataset (40,000 training samples and 1,023 benchmark paragraphs) and a RoBERTa-based baseline model achieving 79.3% F1, outperforming all evaluated methods.Annotation guidelines, dataset, and baseline model were made publicly available.