Quality assurance · Production

Evaluating Deep Agents: LangChain's learnings on test patterns for stateful AI agents

The problem

Traditional LLM evaluation treats every datapoint identically with a shared evaluator, but deep agents require bespoke test logic per datapoint because success criteria vary and involve assertions about trajectory and state beyond just the final message.

First attempt

The naive approach of hardcoding sequential inputs for multi-turn agent tests breaks when the agent deviates from the expected path, making subsequent hardcoded inputs nonsensical, and standard single-evaluator eval pipelines cannot accommodate per-datapoint success criteria.

Workflow diagram · grounded in source
1
Build bespoke test cases
trigger
“Deep agents require bespoke test logic for each datapoint — each test case has its own success criteria”
2
Single-step eval
validation
“Running a deep agent for a single-step is great for validating decision-making in specific scenarios (and saves tokens too!)”
3
Full agent turn eval
validation
“Full agent turns are great for testing assertions about the agent's "end state"”
4
Multi-turn simulation
validation
“Multiple agent turns simulate realistic user interactions but need to be kept on rails”
5
Reproducible environment setup
integration
“Deep Agents need a fresh, clean environment for each eval run in order to ensure reproducible results”
6
Log results to LangSmith
output
“This LangSmith integration automatically logs all test cases to an experiment, so you can view traces for a failed test case (to debug what went wrong) and track results over time”
Reported outcome

LangChain shipped four deep agent applications and distilled five evaluation patterns — bespoke test logic, single-step evals, full agent turns, multi-turn simulation, and reproducible environment setup — integrated with LangSmith for trace logging and result tracking.

Reported stack
LangSmithLangGraphPytestVitestDockerHarborvcrHono
Source
https://blog.langchain.com/evaluating-deep-agents-our-learnings/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LangChain shipped four deep agent applications and distilled five evaluation patterns — bespoke test logic, single-step evals, full agent turns, multi-turn simulation, and reproducible environment setup — integrated w…

What tools did this team use?

LangSmith, LangGraph, Pytest, Vitest, Docker, Harbor, vcr, Hono.

What failed first in this deployment?

The naive approach of hardcoding sequential inputs for multi-turn agent tests breaks when the agent deviates from the expected path, making subsequent hardcoded inputs nonsensical, and standard single-evaluator eval p…

How is this quality assurance AI workflow structured?

Build bespoke test cases → Single-step eval → Full agent turn eval → Multi-turn simulation → Reproducible environment setup → Log results to LangSmith.

Related quality assurance cases