Legal document review · Production

Thomson Reuters CoCounsel: Evaluating Long Context LLM Performance for Legal AI

The problem

Legal documents routinely exceed hundreds of pages, and simply fitting text into a large LLM context window does not guarantee effective performance — the more text included, the higher the risk of missing critical details, and effective context windows are often much smaller than advertised limits.

First attempt

Early GPT-4 had only an 8K token context window requiring document chunking, and RAG underperforms for complex legal queries that require comparing information across an entire document because semantic retrieval returns only passages explicitly matching the query.

Workflow diagram · grounded in source
1
User uploads legal document
trigger
“Users can upload documents, and CoCounsel can automatically perform various tasks on these documents, saving lawyers valuable time”
2
Long context LLM analysis
ai_action
“in CoCounsel 2.0 we leverage long context LLMs to the greatest extent possible to ensure all relevant context is passed to the LLM”
3
Initial benchmark screening
validation
“We use over 20,000 test samples from open and private benchmarks covering legal reasoning, contract understanding, hallucinations, instruction following, and long context capability”
4
Skill-specific SME evaluation
validation
“The top-performing LLMs from our initial benchmarks are tested on our actual skills. This stage involves iteratively developing (sometimes very complex) prompt flows specific to each skill to ensure the LLM consistently generates accurat…”
5
Attorney SME manual review
human_review
“All new LLMs undergo rigorous manual review by our attorney SMEs before deployment. Our SMEs can capture nuanced details missed by automated graders and provide feedback to our engineers for improvement. These SMEs further provide the fi…”
Reported outcome

CoCounsel 2.0 leverages long context LLMs to the greatest extent possible, backed by a multi-stage evaluation pipeline using over 20,000 benchmark test samples and final manual review by attorney SMEs, saving lawyers valuable time on document-centric legal tasks.

Reported metrics
Benchmark test samples used in evaluationover 20,000
Time saved for lawyerssaving lawyers valuable time
Long context test input length range100K–1M tokens
Reported stack
CoCounselGPT-4GPT-4.1o1-miniWestlawReuters NewsNovelQARAG
Source
https://www.thomsonreuters.com/en-us/posts/innovation/legal-ai-benchmarking-evaluating-long-context-performance-for-llms/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

CoCounsel 2.0 leverages long context LLMs to the greatest extent possible, backed by a multi-stage evaluation pipeline using over 20,000 benchmark test samples and final manual review by attorney SMEs, saving lawyers…

What tools did this team use?

CoCounsel, GPT-4, GPT-4.1, o1-mini, Westlaw, Reuters News, NovelQA, RAG.

What results were reported?

Benchmark test samples used in evaluation: over 20,000; Time saved for lawyers: saving lawyers valuable time; Long context test input length range: 100K–1M tokens (source-reported, not independently verified).

What failed first in this deployment?

Early GPT-4 had only an 8K token context window requiring document chunking, and RAG underperforms for complex legal queries that require comparing information across an entire document because semantic retrieval retu…

How is this legal document review AI workflow structured?

User uploads legal document → Long context LLM analysis → Initial benchmark screening → Skill-specific SME evaluation → Attorney SME manual review.

Related legal document review cases