Incident management · Production

Cleric uses LangSmith to run concurrent AI investigations and generalize learnings across deployments

The problem

Engineering teams face complex, time-consuming production issue debugging that drains productivity. The Cleric team specifically needed to test different investigation approaches simultaneously but had no clear way to monitor, compare, or determine which strategies would work best for different issue types — and no mechanism to generalize successful learnings across customer deployments without leaking environment-specific details.

Workflow diagram · grounded in source
1
Alert fires, investigation starts
trigger
“When an alert fires, Cleric automatically begins investigating using existing observability tools and infrastructure”
2
Parallel system examination
ai_action
“Cleric examines multiple systems simultaneously - checking database metrics, network traffic, application logs, and system resources through read-only access to production systems. This parallel investigation approach helps quickly ident…”
3
Findings shared via Slack
output
“Cleric communicates with teams through Slack, sharing its findings and asking for guidance when needed”
4
Concurrent strategies tracked in LangSmith
integration
“LangSmith helped to address this problem by providing clear visibility into these parallel investigations and experiments. With LangSmith, the Cleric system can now: Compare different investigation strategies side-by-side”
5
Engineer feedback provided
human_review
“engineers provide feedback through their normal interactions with Cleric (Slack, ticketing systems, etc.)”
6
Feedback captured via LangSmith API
integration
“This feedback is captured through LangSmith's feedback API and tied directly to the investigation trace”
7
Patterns generalized to shared memories
ai_action
“The system analyzes these patterns to create generalized memories that strip away environment specific details while preserving the core problem-solving approach”
8
Generalized memories deployed across customers
feedback_loop
“These generalized memories are then made available selectively during new investigations across all deployments. LangSmith helps track when and how these memories are used, and whether they improve investigation outcomes”
Reported outcome

LangSmith provided visibility into parallel investigations, enabling side-by-side strategy comparison, feedback capture tied to specific traces, and data-driven validation of which approaches lead to faster resolutions.
Cleric now maintains separate knowledge spaces — customer-specific context alongside a growing library of generalized problem-solving patterns available across all deployments.

Reported metrics
Investigation resolution speedfaster resolutions
Investigation success rates and resolution times trackedinvestigation success rates, resolution times, and other key metrics
Reported stack
ClericLangSmithSlackticketing systems
Source
https://blog.langchain.dev/customers-cleric/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LangSmith provided visibility into parallel investigations, enabling side-by-side strategy comparison, feedback capture tied to specific traces, and data-driven validation of which approaches lead to faster resolutions.

What tools did this team use?

Cleric, LangSmith, Slack, ticketing systems.

What results were reported?

Investigation resolution speed: faster resolutions; Investigation success rates and resolution times tracked: investigation success rates, resolution times, and other key metrics (source-reported, not independently verified).

How is this incident management AI workflow structured?

Alert fires, investigation starts → Parallel system examination → Findings shared via Slack → Concurrent strategies tracked in LangSmith → Engineer feedback provided → Feedback captured via LangSmith API → Patterns generalized to shared memories → Generalized memories deployed across customers.

Related incident management cases