Quality assurance · Production

Shopify introduces Roast: an open-source convention-oriented framework for structured, reproducible AI workflows at developer scale

The problem

Unrestricted AI agents working across millions of lines of code were unreliable due to non-determinism, blocking Shopify from addressing developer problems like flaky tests and insufficient test coverage at scale with minimal human intervention.

First attempt

Unconstrained AI agents could not reliably traverse large codebases; they needed discrete structured steps to stay on track.

Workflow diagram · grounded in source
1
Workflow defined in YAML
trigger
“Simply create a workflow.yml file and corresponding prompt files, and you're ready to go”
2
Deterministic steps execute
integration
“we use deterministic steps to clean up the code and run Sorbet's autocorrect, then hand off remaining issues to the CodingAgent”
3
AI analyzes or generates
ai_action
“Whether you're analyzing code quality, generating documentation, or anything else that involves a series of AI and non-AI steps, Roast provides the structure to make the AI parts work reliably at scale”
4
CodingAgent iterates on complex tasks
ai_action
“It will iteratively fix type errors, run tests, and ensure everything passes—something that would be nearly impossible with pure deterministic automation”
5
Session replay enables resumption
feedback_loop
“Every workflow execution is automatically saved, allowing you to resume from any step”
6
Comprehensive report produced
output
“You get a comprehensive report”
Reported outcome

Since deploying Roast internally, Shopify engineers have analyzed thousands of test files, significantly increased test coverage, automated Sorbet type annotation, enabled proactive SRE monitoring, and saved hours of manual competitive research, while a dozen community engineers have contributed to the open-source project.

Reported metrics
Test files analyzedthousands of test files
Test coverage improvementsignificantly increasing test coverage across the board
Manual research time savedsaving hours of manual research
Community contributorsa dozen Engineers
Reported stack
RoastClaude CodeRaixSorbetRubySlackOpenAI APIOpenRouter
Source
https://shopify.engineering/introducing-roast
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Since deploying Roast internally, Shopify engineers have analyzed thousands of test files, significantly increased test coverage, automated Sorbet type annotation, enabled proactive SRE monitoring, and saved hours of…

What tools did this team use?

Roast, Claude Code, Raix, Sorbet, Ruby, Slack, OpenAI API, OpenRouter.

What results were reported?

Test files analyzed: thousands of test files; Test coverage improvement: significantly increasing test coverage across the board; Manual research time saved: saving hours of manual research; Community contributors: a dozen Engineers (source-reported, not independently verified).

What failed first in this deployment?

Unconstrained AI agents could not reliably traverse large codebases; they needed discrete structured steps to stay on track.

How is this quality assurance AI workflow structured?

Workflow defined in YAML → Deterministic steps execute → AI analyzes or generates → CodingAgent iterates on complex tasks → Session replay enables resumption → Comprehensive report produced.

Related quality assurance cases