Quality assurance · Production

Block builds Actionable CI with three-layer LLM and agentic autofix pipeline

The problem

Block's CI pipelines produce overwhelming failure output across thousands of engineers and large interconnected repositories, making the bottleneck not running CI but understanding what failed and why.

Workflow diagram · grounded in source

CI build failure detected

trigger

“When a CI build fails, the first step is entirely deterministic.”

Static analysis scan

validation

“We scan artifacts from failed jobs using static analysis rules that match common, well-known failure patterns. These are the known knowns. Dependency conflicts, import violations, common configuration mistakes. They are fast, reliable, a…”

LLM log analysis and grouping

ai_action

“bundle the logs from failing jobs and send them to a language model with clear instructions to: Identify the distinct issues present in the logs, Explain each issue in plain language, with awareness of the code changes on the branch, Gro…”

Autofix eligibility check

validation

“we check whether every detected issue is a supported type and the total count is under a threshold. If any issue is not a good candidate, we skip autofix entirely.”

Goose agent fix generation

ai_action

“The analysis from the first two layers is sent to a headless instance of Goose, with instructions to generate a fix.”

Draft PR CI validation loop

feedback_loop

“Goose opens a draft pull request with the proposed changes. CI runs on the draft PR. If it passes, we move forward. If it fails, Goose analyzes the new failure and retries, up to a configured limit.”

Promotion or graceful exit

routing

“If the PR passes and the original branch has not changed, the draft is promoted to ready for review and the developer is notified that their build has been fixed. If the developer already pushed their own fix, the autofix PR is quietly c…”

Reported outcome

Actionable CI surfaces explanations, root causes, and automated code fixes directly in the CI results page, grouping multiple failure symptoms into single issues and enabling both developers and AI coding agents to resolve CI failures.

Reported metrics

Issue grouping benefit15 problems versus one with grouping

Reported stack

Gooselanguage modelMCP server

Source

https://engineering.block.xyz/blog/actionable-ci

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Actionable CI surfaces explanations, root causes, and automated code fixes directly in the CI results page, grouping multiple failure symptoms into single issues and enabling both developers and AI coding agents to re…

What tools did this team use?

Goose, language model, MCP server.

What results were reported?

Issue grouping benefit: 15 problems versus one with grouping (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

CI build failure detected → Static analysis scan → LLM log analysis and grouping → Autofix eligibility check → Goose agent fix generation → Draft PR CI validation loop → Promotion or graceful exit.