Quality assurance · Production

Factory uses LangSmith to achieve 2x iteration speed and 20% cycle time reduction in SDLC automation

The problem

Factory's LLM-based Droids needed robust observability in customer environments with tight data controls, while existing tools were cumbersome for tracking data flow and debugging context-awareness issues. Manual prompt optimization was also time-consuming and inaccurate.

Workflow diagram · grounded in source

Droid SDLC workflow initiates

trigger

“Factory's fleet of Droids automate different stages of the SDLC, boosting engineering velocity for large organizations”

Trace export to CloudWatch

integration

“Factory integrated LangSmith to export traces to AWS CloudWatch logs, which allowed the team to precisely track data flow through various stages of the LLM pipeline”

Agentic step pinpointing

validation

“By linking LangSmith events and steps with CloudWatch logs, Factory's engineers could pinpoint their position in the agentic stage. This integration helped maintain a single source of truth for data flow in LLM from one step to the next,…”

Feedback linked to LLM calls

feedback_loop

“Factory used LangSmith to link feedback directly to each LLM call, providing immediate insights into potential problems. This integration helped the team quickly identify and resolve issues like hallucinations”

Feedback exported and analyzed

feedback_loop

“The feedback was then exported to datasets, and analyzed for patterns and areas for improvement”

LLM prompt self-analysis

ai_action

“they had the LLM look at a prompt and make a claim as to why the prompt may have caused a bad example (and not a good example)”

Reported outcome

Factory achieved 2x iteration speed, an average ~20% reduction in open-to-merge time, and a 3x reduction in code churn in the first 90 days.
Clients also report an average cycle time reduction of up to 20% and over 550,000 hours of development time saved.

Reported metrics

Iteration speed2x

Open-to-merge time reduction~20%

Code churn reduction3x

Cycle time reductionup to 20%

Show all 5 reported metrics

iteration speed2x

open-to-merge time reduction~20%

code churn reduction3x

cycle time reductionup to 20%

development time savedover 550,000 hours

Reported stack

LangSmithLangChainAWS CloudWatchFeedback API

Source

https://blog.langchain.dev/customers-factory/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Factory achieved 2x iteration speed, an average ~20% reduction in open-to-merge time, and a 3x reduction in code churn in the first 90 days.

What tools did this team use?

LangSmith, LangChain, AWS CloudWatch, Feedback API.

What results were reported?

Iteration speed: 2x; Open-to-merge time reduction: ~20%; Code churn reduction: 3x; Cycle time reduction: up to 20% (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Droid SDLC workflow initiates → Trace export to CloudWatch → Agentic step pinpointing → Feedback linked to LLM calls → Feedback exported and analyzed → LLM prompt self-analysis.

Factory achieved 2x iteration speed, an average ~20% reduction in open-to-merge time, and a 3x reduction in code churn in the first 90 days.Clients also report an average cycle time reduction of up to 20% and over 550,000 hours of development time saved.