Quality assurance · Production

Trunk engineering lessons for building reliable AI agents for CI root cause analysis

The problem

LLM nondeterminism makes it difficult to build reliable AI agents for DevOps/CI tasks—the same inputs can produce different outputs, making testing and consistent user experience challenging.

First attempt

Extensive prompt engineering to make Claude call tools in a deterministic manner failed; switching to Gemini resolved the issue at the cost of some LLM reasoning quality.

Workflow diagram · grounded in source
1
CI failure triggers RCA
trigger
“we started by focusing on root cause analysis (RCA) for test failures”
2
Retrieve historical stack traces
integration
“Our existing Flaky Tests feature stores historical stack traces from test failures in CI, so we already have a rich dataset and existing infrastructure to support this use case”
3
Preprocess CI log data
validation
“We group similar failures and remove non-relevant info so we stay within our LLM's context limit when processing CI logs”
4
Agent examines failures
ai_action
“add an agent into the stack that examines failures and posts summaries to GitHub PRs”
5
Subagent structured extraction
ai_action
“Subagents are used to extract relevant, structured outputs, summarize them, and funnel them to the user”
6
Deterministic output validation
validation
“If the agent fails to meet deterministic output validation, like surpassing a character limit, we can rerun it”
7
Post summaries to GitHub PRs
output
“posts summaries to GitHub PRs”
8
Developer feedback collection
feedback_loop
“we have a feedback form attached to the PR comments left by the agent so devs can leave feedback, both positive and negative”
Reported outcome

Trunk built an AI agent for CI root cause analysis that produces better output and more reliable tests, enabling incremental improvements and described as a massive speed boost for previously manual tasks.

Reported metrics
Agent output qualitybetter output
Speed boost from partial automationmassive speed boost for manual and repetitive tasks
Test reliabilitymore reliable tests
Reported stack
LangSmithMSWVercel's AI SDKClaudeGeminiGitHub
Source
https://trunk.io/blog/attempting-to-engineer-the-chaos-out-of-ai-agents
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Trunk built an AI agent for CI root cause analysis that produces better output and more reliable tests, enabling incremental improvements and described as a massive speed boost for previously manual tasks.

What tools did this team use?

LangSmith, MSW, Vercel's AI SDK, Claude, Gemini, GitHub.

What results were reported?

Agent output quality: better output; Speed boost from partial automation: massive speed boost for manual and repetitive tasks; Test reliability: more reliable tests (source-reported, not independently verified).

What failed first in this deployment?

Extensive prompt engineering to make Claude call tools in a deterministic manner failed; switching to Gemini resolved the issue at the cost of some LLM reasoning quality.

How is this quality assurance AI workflow structured?

CI failure triggers RCA → Retrieve historical stack traces → Preprocess CI log data → Agent examines failures → Subagent structured extraction → Deterministic output validation → Post summaries to GitHub PRs → Developer feedback collection.

Related quality assurance cases