quality_assurance · saas · workflow
Shopify introduces Roast: an open-source convention-oriented framework for structured, reproducible AI workflows at developer scale
Unrestricted AI agents working across millions of lines of code were unreliable due to non-determinism, blocking Shopify from addressing developer problems like flaky tests and insufficient test coverage at scale with minimal human intervention.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Workflow defined in YAML
A developer creates a workflow.yml file and corresponding prompt files to launch a structured AI workflow.
Tools used
RoastClaude CodeRaixSorbetRubySlack
Outcome
Since deploying Roast internally, Shopify engineers have analyzed thousands of test files, significantly increased test coverage, automated Sorbet type annotation, enabled proactive SRE monitoring, and saved hours of manual competitive research, while a dozen community engineers have contributed to the open-source project.
What failed first
Unconstrained AI agents could not reliably traverse large codebases; they needed discrete structured steps to stay on track.
Results
Time savedsaving hours of manual research
Volumea dozen Engineers
Running sincethis past year
Grounding & classification
Source type: technical build writeup
32 fields verified against source quotes, 2 dropped as unverifiable.
agentic workflowai agentcode generationquality inspectionsummarizationcode diff prbuilder submittedfailure mode describedmetric backednamed customerproduction runtime claimedtools describedworkflow describedsoftwareaccuracy improvementemployee productivitythroughput increasetechnical build writeupincident managementquality assuranceagentic task execution