Quality assurance · Production

Replit uses LangSmith to gain observability into Replit Agent's complex agentic workflows

The problem

Replit Agent's complex multi-step workflows generated traces involving hundreds of steps, posing significant challenges for ingestion, display, and debugging; searching within long traces was not possible, and multi-turn conversational sessions produced disjoint traces that were difficult to correlate.

Workflow diagram · grounded in source
1
Replit Agent executes tasks
trigger
“performs a wider range of functions – including planning, creating dev environments, installing dependencies, and deploying applications for users”
2
LangSmith traces execution
integration
“It captures multiple LLM calls as well as other steps (retrieval, running code, etc). This gives you granular visibility into what's happening, including at the inputs and outputs of each step, in order to understand the agent's decision…”
3
Search within traces
validation
“users could now filter directly on a criteria they cared about (e.g. keywords in the inputs or outputs of a run)”
4
Thread view collates sessions
integration
“LangSmith's thread view helped collate traces from multiple threads together that were related (i.e. from one conversation). This provided a logical view of all agent-user interactions across a multi-turn conversation”
5
Human developer correction
human_review
“human developers, who can come in and edit and correct agent trajectories as needed”
Reported outcome

LangChain improved LangSmith's ingestion and rendering for large traces, added search-within-traces functionality, and introduced a thread view for multi-turn conversations — greatly reducing Replit's debugging time and speeding up the process of building and scaling complex agents.

Reported metrics
Time to debug agent steps within a tracegreatly reduced
Speed of building and scaling complex agentsgreatly sped up
Reported stack
LangSmith
Source
https://blog.langchain.dev/customers-replit/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LangChain improved LangSmith's ingestion and rendering for large traces, added search-within-traces functionality, and introduced a thread view for multi-turn conversations — greatly reducing Replit's debugging time a…

What tools did this team use?

LangSmith.

What results were reported?

Time to debug agent steps within a trace: greatly reduced; Speed of building and scaling complex agents: greatly sped up (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Replit Agent executes tasks → LangSmith traces execution → Search within traces → Thread view collates sessions → Human developer correction.

Related quality assurance cases