legal_document_review · saas · workflow
Wordsmith uses LangSmith for LLM observability, enabling 10x cost reduction on inference tasks and debug time from minutes to seconds
As Wordsmith's LLM-powered features grew exponentially, the engineering team lacked visibility into LLM performance and interactions in production and relied on CloudWatch logs to debug complex multi-stage inference chains, which was slow and painful.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Multi-source data ingestion
Wordsmith ingests Slack messages, Zendesk tickets, pull requests, and legal documents across a heterogeneous set of domains.
Tools used
LangSmithStatsigGPT-4Claude 3.5GPT-4oOpenAIAnthropicGoogleMistralCloudwatch
Outcome
LangSmith reduced inference debug time from minutes to seconds, allowed the team to compare and deploy a new model within an hour, and enabled cost reductions of up to 10x on particular inference tasks by facilitating model selection optimization.
What failed first
Wordsmith's engineering team relied solely on CloudWatch logs for debugging, which proved too slow and painful for the complex nested inference chains in their production system.
Results
Time savedfrom minutes to seconds
Cost replacedup to 10x
Grounding & classification
Source type: vendor customer story
39 fields verified against source quotes.
agentic workflowcontent generationdocument airagcode diff prcontractemailknowledge basesupport ticketfailure mode describedmetric backednamed customerproduction runtime claimedtools describedworkflow describedlegalsoftwarecost reductioncycle time reductionemployee productivityvendor customer storycontract managementlegal document reviewlegal opsagentic task executionrag answering