Compliance monitoring · Production

Dropbox discovers and discloses repeated-token training data extraction vulnerability in GPT-3.5 and GPT-4

The problem

OpenAI's GPT-3.5 and GPT-4 models were vulnerable to divergence attacks triggered by repeated token sequences, allowing extraction of memorized training data including PII and sensitive information. After OpenAI deployed an initial mitigation filtering single-token repeats, multi-token repeat sequences still allowed exploitation of the same vulnerability.

First attempt

OpenAI's initial filtering defense focused exclusively on single-token repetitions (reflecting the emphasis of the Scalable Extraction paper), leaving multi-token repeat sequences able to induce the same model divergence and training data extraction from both GPT-3.5 and GPT-4.

Workflow diagram · grounded in source
1
Internal AI security review
trigger
“In April 2023, while performing an internal security review of our AI-powered products, Dropbox discovered a ChatGPT prompt injection vulnerability triggered by the presence of repeated character sequences in user-controlled portions of …”
2
Repeated multi-token divergence attack
ai_action
“Dropbox used repeated multi-token (>1) sequences to induce divergence in ChatGPT models and demonstrated extraction of memorized training data from both GPT-3.5 and GPT-4”
3
Training data match validation
validation
“Note that the text from the beginning of the GPT-3.5 response can be found verbatim within the jq documentation hosted on Github”
4
GPT-4 memorized passage extraction
ai_action
“When the phrase is repeated 12,000 times and sent in a gpt-4-32k prompt, the model responds with an excerpt from the Old Testament's Book of Genesis (specifically, Chapter 24, verses 16-27)”
5
Responsible disclosure to OpenAI
integration
“we formally shared our results with OpenAI via disclosure@openai.com on January 24, 2024”
6
Remediation and research publication
output
“OpenAI updated their filtering strategy to block prompts with multi-token repeats on January 29, 2024...OpenAI subsequently allowed us to publish this research inside their requested disclosure period”
Reported outcome

After Dropbox's responsible disclosure in January 2024, OpenAI confirmed the training data extraction vulnerabilities and patched them by extending filtering to block multi-token repeat prompts and adding a server-side timeout for long-running requests.

Reported metrics
sequential matching tokens in GPT-3.5 training data extractionover 100 sequential cl100k_base tokens
GPT-4 streaming response tokens before timeoutover 10,700 tokens
non-streaming GPT-4 request timeout durationten minutes
estimated full GPT-4 long-running request durationthirty minutes
Reported stack
GPT-3.5GPT-4PythonLangchain
Source
https://dropbox.tech/machine-learning/bye-bye-bye-evolution-of-repeated-token-attacks-on-chatgpt-models
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After Dropbox's responsible disclosure in January 2024, OpenAI confirmed the training data extraction vulnerabilities and patched them by extending filtering to block multi-token repeat prompts and adding a server-sid…

What tools did this team use?

GPT-3.5, GPT-4, Python, Langchain.

What results were reported?

sequential matching tokens in GPT-3.5 training data extraction: over 100 sequential cl100k_base tokens; GPT-4 streaming response tokens before timeout: over 10,700 tokens; non-streaming GPT-4 request timeout duration: ten minutes; estimated full GPT-4 long-running request duration: thirty minutes (source-reported, not independently verified).

What failed first in this deployment?

OpenAI's initial filtering defense focused exclusively on single-token repetitions (reflecting the emphasis of the Scalable Extraction paper), leaving multi-token repeat sequences able to induce the same model diverge…

How is this compliance monitoring AI workflow structured?

Internal AI security review → Repeated multi-token divergence attack → Training data match validation → GPT-4 memorized passage extraction → Responsible disclosure to OpenAI → Remediation and research publication.

Related compliance monitoring cases