Quality assurance · Production

Factory's anchored iterative summarization outperforms OpenAI and Anthropic context compression strategies for long-running AI agent sessions

The problem

Long-running AI agent sessions generate millions of tokens that exceed any model's working memory, and naive aggressive compression causes agents to forget critical details—file paths, error messages, past decisions—leading to wasted tokens re-reading files and re-exploring dead ends.

First attempt

Generic summarization treats all content as equally compressible, silently dropping file paths and decisions; traditional metrics like ROUGE or embedding similarity failed to capture whether an agent can actually continue working after compression.

Workflow diagram · grounded in source
1
Context limit reached
trigger
“Long-running agent sessions can generate millions of tokens of conversation history. That far exceeds what any model can hold in working memory.”
2
Truncated span summarized
ai_action
“When compression triggers, only the newly-truncated span is summarized and merged with the existing summary. We call this anchored iterative summarization.”
3
Merge into persistent summary
integration
“Factory maintains a structured, persistent summary with explicit sections for different information types: session intent, file modifications, decisions made, and next steps.”
4
Checklist sections enforce preservation
output
“By dedicating sections to specific information types, the summary cannot silently drop file paths or skip over decisions. Each section acts as a checklist: the summarizer must populate it or explicitly leave it empty.”
5
Probe-based evaluation
validation
“after compression, ask the agent questions that require remembering specific details from the truncated history. If the compression preserved the right information, the agent answers correctly. If not, it guesses or hallucinates.”
6
LLM judge grades responses
ai_action
“We grade responses using an LLM judge (GPT-5.2) across six dimensions”
Reported outcome

Factory's structured summarization scores 0.35 points higher than OpenAI and 0.26 higher than Anthropic overall, with accuracy showing the largest gap (Factory 4.04), while maintaining comparable compression efficiency (98.6% vs OpenAI's 99.3%).

Reported metrics
Factory overall quality score3.70
Anthropic overall quality score3.44
OpenAI overall quality score3.35
Factory quality advantage over OpenAI0.35 points higher
Show all 15 reported metrics
Factory overall quality score3.70
Anthropic overall quality score3.44
OpenAI overall quality score3.35
Factory quality advantage over OpenAI0.35 points higher
Factory quality advantage over Anthropic0.26 points higher
Factory accuracy score4.04
OpenAI accuracy score3.43
OpenAI compression ratio99.3%
Anthropic compression ratio98.7%
Factory compression ratio98.6%
Dataset size36,611 messages
Factory example recall probe score4.8/5
Factory artifact probe score3.6
OpenAI artifact probe score2.8
GPT-4 agreement with human preferencesover 80%
Reported stack
GPT-5.2Claude SDK/responses/compact
Source
https://factory.ai/news/evaluating-compression
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Factory's structured summarization scores 0.35 points higher than OpenAI and 0.26 higher than Anthropic overall, with accuracy showing the largest gap (Factory 4.04), while maintaining comparable compression efficienc…

What tools did this team use?

GPT-5.2, Claude SDK, /responses/compact.

What results were reported?

Factory overall quality score: 3.70; Anthropic overall quality score: 3.44; OpenAI overall quality score: 3.35; Factory quality advantage over OpenAI: 0.35 points higher (source-reported, not independently verified).

What failed first in this deployment?

Generic summarization treats all content as equally compressible, silently dropping file paths and decisions; traditional metrics like ROUGE or embedding similarity failed to capture whether an agent can actually cont…

How is this quality assurance AI workflow structured?

Context limit reached → Truncated span summarized → Merge into persistent summary → Checklist sections enforce preservation → Probe-based evaluation → LLM judge grades responses.

Related quality assurance cases