Workflow · Production

Dropbox security research: prompt injection via control characters in GPT-3.5 and GPT-4

The problem

Dropbox's security team identified that user-controlled control characters in LLM prompt inputs can circumvent system-level instructions, enabling prompt injection attacks that cause models to betray context constraints or hallucinate.

First attempt

The prompt template designed to constrain LLM queries to a specific context and prevent instruction leakage was defeated when sufficient control characters were prepended to the question parameter, regardless of instruction wording or formatting.

Workflow diagram · grounded in source
1
LLM injection threat identified
trigger
“Injection attacks that manipulate inputs used in LLM queries have been one such focus for Dropbox security engineers”
2
Prompt template constructed
output
“Dropbox experimented with a prompt template similar to that shown below”
3
Control character injection scripted
trigger
“we scripted some repeatable blackbox experiments to learn more about how the GPT-3.5 model interprets them”
4
LLM processes injected prompt
ai_action
“The code invokes OpenAI's /v1/chat/completions endpoint using the gpt-3.5-turbo model”
5
Instruction betrayal observed
validation
“we see the model completely ignore its instructions and forget the context at an offset of -1024. GPT-3.5 hallucinates at offset -3500”
6
Sanitization guidance recommended
output
“the best approach to mitigation involves sanitizing input appropriately for the input and chosen model”
Reported outcome

Dropbox demonstrated that prepending sufficient control characters to LLM prompt inputs causes GPT-3.5 and GPT-4 to betray their system instructions and hallucinate; the team shared findings with OpenAI and identified input sanitization as the primary mitigation.

Reported metrics
Carriage returns required to trigger model instruction betrayal350 or more
backspaces required to trigger GPT-3.5 instruction betrayalat least 450
GPT-3.5 context window token limit4096
GPT-4 context window extension factorfactor of four
Show all 5 reported metrics
carriage returns required to trigger model instruction betrayal350 or more
backspaces required to trigger GPT-3.5 instruction betrayalat least 450
GPT-3.5 context window token limit4096
GPT-4 context window extension factorfactor of four
GPT-4 extended context window tokens32768
Reported stack
GPT-3.5GPT-4ChatGPTPython 3
Source
https://dropbox.tech/machine-learning/prompt-injection-with-control-characters-openai-chatgpt-llm
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Dropbox demonstrated that prepending sufficient control characters to LLM prompt inputs causes GPT-3.5 and GPT-4 to betray their system instructions and hallucinate; the team shared findings with OpenAI and identified…

What tools did this team use?

GPT-3.5, GPT-4, ChatGPT, Python 3.

What results were reported?

Carriage returns required to trigger model instruction betrayal: 350 or more; backspaces required to trigger GPT-3.5 instruction betrayal: at least 450; GPT-3.5 context window token limit: 4096; GPT-4 context window extension factor: factor of four (source-reported, not independently verified).

What failed first in this deployment?

The prompt template designed to constrain LLM queries to a specific context and prevent instruction leakage was defeated when sufficient control characters were prepended to the question parameter, regardless of instr…

How is this workflow AI workflow structured?

LLM injection threat identified → Prompt template constructed → Control character injection scripted → LLM processes injected prompt → Instruction betrayal observed → Sanitization guidance recommended.