Cleric uses LangSmith to run concurrent AI investigations and generalize learnings across deployments
Engineering teams face complex, time-consuming production issue debugging that drains productivity. The Cleric team specifically needed to test different investigation approaches simultaneously but had no clear way to monitor, compare, or determine which strategies would work best for different issue types — and no mechanism to generalize successful learnings across customer deployments without leaking environment-specific details.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Alert fires, investigation starts
When an alert fires, Cleric automatically begins investigating using existing observability tools and infrastructure.
Tools used
ClericLangSmith
Outcome
LangSmith provided visibility into parallel investigations, enabling side-by-side strategy comparison, feedback capture tied to specific traces, and data-driven validation of which approaches lead to faster resolutions. Cleric now maintains separate knowledge spaces — customer-specific context alongside a growing library of generalized problem-solving patterns available across all deployments.
Results
Time savedinvestigation success rates, resolution times, and other key metrics