Quality assurance · Production

Chroma context rot study: LLM performance degrades non-uniformly with increasing context length

The problem

The Needle in a Haystack (NIAH) benchmark is widely used to assert that LLMs handle long contexts reliably, but it only tests narrow lexical retrieval and does not reflect real-world tasks requiring semantic understanding, ambiguity resolution, or distractor handling. Models are consequently assumed to perform uniformly across long-context scenarios when they do not.

First attempt

NIAH produced consistently high scores across all major models, leading to the widely held perception that long-context handling was largely a solved problem—when in fact the benchmark only measured a narrow lexical retrieval capability.

Workflow diagram · grounded in source
1
Controlled experiment design
trigger
“our experiments hold task complexity constant while varying only the input length—allowing us to directly measure the effect of input length alone”
2
Haystack chunk embedding
ai_action
“Chunk documents into 1-3 sentence chunks”
3
HDBSCAN topic clustering
ai_action
“Use HDBSCAN to create clusters with the following parameters: min_cluster_size=10, min_samples=15”
4
Vector DB fairness validation
validation
“We store our previously computed haystack chunk embeddings in a vector database. Query top-10 results from that vector database with our question embedding. Manually examine these results to verify that they do not answer the given quest…”
5
Multi-LLM evaluation runs
ai_action
“We evaluate each model across its maximum context window with temperature=0”
6
GPT-4.1 output judging
validation
“We evaluate model outputs using an aligned GPT-4.1 judge”
7
Performance degradation findings
output
“Across all experiments, model performance consistently degrades with increasing input length.”
Reported outcome

Across all 18 LLMs and experiments, model performance consistently degrades with increasing input length in non-uniform ways.
Distractors amplify degradation as context grows; shuffled haystacks outperform structurally coherent ones; and models perform better when the needle is semantically distinct from the haystack.

Reported metrics
LLMs evaluated18
NoLiMa pairs requiring external knowledge72.4%
LLM call refusal rate0.035%
GPT-4.1 judge alignment to human judgment>99%
Show all 7 reported metrics
LLMs evaluated18
NoLiMa pairs requiring external knowledge72.4%
LLM call refusal rate0.035%
GPT-4.1 judge alignment to human judgment>99%
LongMemEval prompts evaluated after filtering306
Focused prompt average token length~300 tokens
Full LongMemEval prompt token length~113k tokens
Reported stack
text-embedding-3-smalltext-embedding-3-largejina-embeddings-v3voyage-3-largeall-MiniLM-L6-v2LongMemEvalvector databaseGPT-4.1Llama 4Claude Opus 4Claude Sonnet 4GPT-3.5 turbo
Source
https://research.trychroma.com/context-rot
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Across all 18 LLMs and experiments, model performance consistently degrades with increasing input length in non-uniform ways.

What tools did this team use?

text-embedding-3-small, text-embedding-3-large, jina-embeddings-v3, voyage-3-large, all-MiniLM-L6-v2, LongMemEval, vector database, GPT-4.1, Llama 4, Claude Opus 4.

What results were reported?

LLMs evaluated: 18; NoLiMa pairs requiring external knowledge: 72.4%; LLM call refusal rate: 0.035%; GPT-4.1 judge alignment to human judgment: >99% (source-reported, not independently verified).

What failed first in this deployment?

NIAH produced consistently high scores across all major models, leading to the widely held perception that long-context handling was largely a solved problem—when in fact the benchmark only measured a narrow lexical r…

How is this quality assurance AI workflow structured?

Controlled experiment design → Haystack chunk embedding → HDBSCAN topic clustering → Vector DB fairness validation → Multi-LLM evaluation runs → GPT-4.1 output judging → Performance degradation findings.

Related quality assurance cases