Quality assurance · Production

Monday.com builds an evals-driven development framework for its AI service workforce with LangSmith

The problem

Building a ReAct-based AI service workforce introduced cascading quality risks where a minor prompt deviation could compound across multi-step reasoning chains, yet most teams treat evaluation as a last-mile check rather than a Day 0 requirement.

First attempt

Running offline evaluations serially created a major bottleneck in the development loop, forcing a tradeoff between testing depth and development pace.

Workflow diagram · grounded in source
1
Offline eval against golden dataset
validation
“Offline Evaluations — "The Safety Net": Acting somewhat like a unit-testing layer, runs the agent against curated "golden datasets". Tests both core logic (e.g., groundedness, retrieval accuracy, tool-calling) and specific edge cases (e.…”
2
Deterministic smoke checks
validation
“Deterministic "smoke" checks: Runtime health: the agent ran with no crashes/timeouts, request succeeds end-to-end. Output shape: the response matches the expected schema/format (even before judging content). State & persistence: thread/s…”
3
LLM-as-judge semantic scoring
ai_action
“LLM-as-a-Judge: Semantic grading against a "Golden Dataset". We leverage open-source libraries like OpenEvals and AgentEvals to score dimensions like correctness and groundedness.”
4
Parallelized eval execution
validation
“By optimizing our Vitest and LangSmith integration, we achieved a massive speed increase by distributing the load across local workers and remote API calls. The key was a hybrid approach: parallelizing test files to maximize CPU usage an…”
5
Online multi-turn production monitoring
ai_action
“we can now use custom prompts to grade the entire conversation trajectory—measuring high-level outcomes like user satisfaction, tone, and goal resolution”
6
CI/CD eval deployment
integration
“we built a custom CLI command, yarn eval deploy, that runs in our CI/CD pipeline. This ensures our repository remains the absolute Source of Truth for our evaluation infrastructure.”
Reported outcome

Monday.com achieved 8.7x faster evaluation feedback loops (from 162 seconds to 18 seconds), comprehensive coverage across hundreds of examples in minutes instead of hours, real-time production monitoring via Multi-Turn Evaluators, and evaluation logic managed as version-controlled production code with GitOps-style CI/CD deployment.

Reported metrics
Evaluation feedback loop speedup8.7x faster
Sequential eval duration baseline162.35s
Parallel+concurrent eval duration18.60s
Concurrent-only eval speedup4.1x faster
Show all 5 reported metrics
evaluation feedback loop speedup8.7x faster
sequential eval duration baseline162.35s
parallel+concurrent eval duration18.60s
concurrent-only eval speedup4.1x faster
testing coveragehundreds of examples in minutes instead of hours
Reported stack
LangSmithLangGraphVitestDocumentation MCPLangSmith MCPCursorClaude Code
Source
https://blog.langchain.com/customers-monday/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Monday.com achieved 8.7x faster evaluation feedback loops (from 162 seconds to 18 seconds), comprehensive coverage across hundreds of examples in minutes instead of hours, real-time production monitoring via Multi-Tur…

What tools did this team use?

LangSmith, LangGraph, Vitest, Documentation MCP, LangSmith MCP, Cursor, Claude Code.

What results were reported?

Evaluation feedback loop speedup: 8.7x faster; Sequential eval duration baseline: 162.35s; Parallel+concurrent eval duration: 18.60s; Concurrent-only eval speedup: 4.1x faster (source-reported, not independently verified).

What failed first in this deployment?

Running offline evaluations serially created a major bottleneck in the development loop, forcing a tradeoff between testing depth and development pace.

How is this quality assurance AI workflow structured?

Offline eval against golden dataset → Deterministic smoke checks → LLM-as-judge semantic scoring → Parallelized eval execution → Online multi-turn production monitoring → CI/CD eval deployment.

Related quality assurance cases