Sales ops · Production

Five steps for building machine learning models that drive business impact at Shopify

The problem

While technical ML documentation is abundant, very few resources address how machine learning fits within a business context — covering when to start, how to get started, and how to update models over time without breaking the product.

Workflow diagram · grounded in source
1
Assess ML readiness
validation
“the first question to ask is: should I invest resources in a machine learning model at this time?”
2
Establish heuristic baseline
validation
“In 90 percent of cases, you can create a baseline using heuristics”
3
Build simplest initial model
ai_action
“robust, interpretable models that train quickly (shallow decision tree, linear or logistic regression are three good initial choices)”
4
Tie metric to business impact
validation
“The metric should align with the primary objectives of the business”
5
A/B test model iterations
feedback_loop
“An online A/B test works well in most cases. By exposing a random group of users to our new version of the model, we get a clear view of it's impact relative to our baseline”
6
Monitor prediction stability
feedback_loop
“Being aware of this effect and measuring it is the first line of defense”
Reported outcome

Shopify's five-step methodology is described as giving a straightforward workflow to productionize models that drive impact; internal applications include a leads-scoring model where a simpler random forest was kept over a more complex ensemble, and a shop industry classifier that ran in staging for weeks before promotion to production.

Reported metrics
Share of problems where heuristic baseline is achievable90 percent
Share of high-apparent-performance cases masking a model issueabout 95 percent
Reported stack
random forestlogistic regressionshallow decision treeword embedding model
Source
https://shopify.engineering/building-business-machine-learning-models
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Shopify's five-step methodology is described as giving a straightforward workflow to productionize models that drive impact; internal applications include a leads-scoring model where a simpler random forest was kept o…

What tools did this team use?

random forest, logistic regression, shallow decision tree, word embedding model.

What results were reported?

Share of problems where heuristic baseline is achievable: 90 percent; Share of high-apparent-performance cases masking a model issue: about 95 percent (source-reported, not independently verified).

How is this sales ops AI workflow structured?

Assess ML readiness → Establish heuristic baseline → Build simplest initial model → Tie metric to business impact → A/B test model iterations → Monitor prediction stability.

Related sales ops cases