Legal document review · Production

Wordsmith uses LangSmith for LLM observability, enabling 10x cost reduction on inference tasks and debug time from minutes to seconds

The problem

As Wordsmith's LLM-powered features grew exponentially, the engineering team lacked visibility into LLM performance and interactions in production and relied on CloudWatch logs to debug complex multi-stage inference chains, which was slow and painful.

First attempt

Wordsmith's engineering team relied solely on CloudWatch logs for debugging, which proved too slow and painful for the complex nested inference chains in their production system.

Workflow diagram · grounded in source
1
Multi-source data ingestion
trigger
“Wordsmith ingests Slack messages, Zendesk tickets, pull requests, and legal documents, delivering accurate results over a heterogeneous set of domains and NLP tasks”
2
RAG and multistage LLM inference
ai_action
“Wordsmith's first feature was a configurable RAG pipeline for Slack. It has now evolved to support complex multistage inferences over a wide variety of data sources and objectives”
3
Draft response delivered to legal team
output
“integrates seamlessly into email and messaging systems to automatically draft responses for the legal team, mimicking what it's like to work with another person on the team”
4
LangSmith hierarchical tracing
integration
“These workflows can contain up to 100 nested inferences, making it time-consuming and painful to sift through general logs to find the root cause of an errant response. With LangSmith's out-of-the-box tracing interface, diagnosing poor p…”
5
Evaluation sets validate model performance
validation
“Wordsmith has published a variety of evaluation sets for various tasks like RAG, agentic workloads, attribute extractions, and even XML-based changeset targeting — facilitating their deployment to production. These static evaluation sets…”
6
Statsig experiment tags mapped to LangSmith
feedback_loop
“Wordsmith uses Statsig as their feature flag / experiment exposure library. Leveraging LangSmith tags, it's simple to map each exposure to the appropriate tag in LangSmith for simplified experiment analyses”
Reported outcome

LangSmith reduced inference debug time from minutes to seconds, allowed the team to compare and deploy a new model within an hour, and enabled cost reductions of up to 10x on particular inference tasks by facilitating model selection optimization.

Reported metrics
Cost reduction on particular inference tasksup to 10x
Inference debug timefrom minutes to seconds
Model evaluation and deployment timewithin an hour
Reported stack
LangSmithStatsigGPT-4Claude 3.5GPT-4oOpenAIAnthropicGoogleMistralCloudwatchSlackZendesk
Source
https://blog.langchain.dev/customers-wordsmith/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LangSmith reduced inference debug time from minutes to seconds, allowed the team to compare and deploy a new model within an hour, and enabled cost reductions of up to 10x on particular inference tasks by facilitating…

What tools did this team use?

LangSmith, Statsig, GPT-4, Claude 3.5, GPT-4o, OpenAI, Anthropic, Google, Mistral, Cloudwatch.

What results were reported?

Cost reduction on particular inference tasks: up to 10x; Inference debug time: from minutes to seconds; Model evaluation and deployment time: within an hour (source-reported, not independently verified).

What failed first in this deployment?

Wordsmith's engineering team relied solely on CloudWatch logs for debugging, which proved too slow and painful for the complex nested inference chains in their production system.

How is this legal document review AI workflow structured?

Multi-source data ingestion → RAG and multistage LLM inference → Draft response delivered to legal team → LangSmith hierarchical tracing → Evaluation sets validate model performance → Statsig experiment tags mapped to LangSmith.

Related legal document review cases