Quality assurance · Production

Block's Builderbot: AI protection agents for system architecture with parallel code review subagents

The problem

Block's engineering organization faced degenerative patterns where individual teams ship features that are locally rational but globally corrosive, and no single engineer can hold the full system in their head. Existing AI tools acted as assistants or advisors that waited for humans to act, and agents fumbled around in unfamiliar repositories without consistent entrypoints.

First attempt

Many agentic reviewers were limited to a single prompt expected to cover an entire system, which proved insufficient for hyperlocal context, and CI had become the default validation layer rather than catching issues earlier in the development lifecycle.

Workflow diagram · grounded in source
1
Pre-push hook triggers checks
trigger
“just fmt or just test via pre-commit and pre-push hooks before pushing code to a PR”
2
Standardized CLI entrypoint
ai_action
“implementing a single common CLI contract for local development in all of our repositories using Just. This enables our local agents to have a standardized entrypoint to all of the same tools that our CI runs”
3
Hyperlocal context loading
ai_action
“Most agents will automatically load an AGENTS.md file when they start working in a directory, and check for more local AGENTS.md context files as they navigate a system”
4
Parallel specialized subagent review
ai_action
“Each check runs as an isolated subagent with its own context window — a global check for API standards loads different context than a module-level check for PCI compliance in payments/ or security review in auth/. These subagents run in …”
5
Findings aggregated into report
output
“their findings are aggregated into a single review report”
6
Human final approval
human_review
“with humans providing the final stamp of approval rather than the initial analysis”
7
Proactive policy evolution
feedback_loop
“We give our protectors a heartbeat to proactively review incidents, announcements, and messages to consider which deterministic and non-deterministic checks to propose for human review”
Reported outcome

Builderbot sits between builders and systems as a continuous protector, with pre-push checks shifting validation left, parallel specialized subagents aggregating findings into a single review report, and humans providing only the final stamp of approval rather than initial analysis.

Reported metrics
Burden on human reviewersreduces the burden on human reviewers
Issue detection speedfaster to catch issues
Local agent effectivenessmassive impacts for our local agents' ability to make the right changes quickly
Reported stack
BuilderbotJustAGENTS.mdAmpAgent Skillssq agents review
Source
https://engineering.block.xyz/blog/protecting-our-systems-with-intelligence
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Builderbot sits between builders and systems as a continuous protector, with pre-push checks shifting validation left, parallel specialized subagents aggregating findings into a single review report, and humans provid…

What tools did this team use?

Builderbot, Just, AGENTS.md, Amp, Agent Skills, sq agents review.

What results were reported?

Burden on human reviewers: reduces the burden on human reviewers; Issue detection speed: faster to catch issues; Local agent effectiveness: massive impacts for our local agents' ability to make the right changes quickly (source-reported, not independently verified).

What failed first in this deployment?

Many agentic reviewers were limited to a single prompt expected to cover an entire system, which proved insufficient for hyperlocal context, and CI had become the default validation layer rather than catching issues e…

How is this quality assurance AI workflow structured?

Pre-push hook triggers checks → Standardized CLI entrypoint → Hyperlocal context loading → Parallel specialized subagent review → Findings aggregated into report → Human final approval → Proactive policy evolution.

Related quality assurance cases