Back office ops · Production

Shopify Merlin Online Inference: deploying ML models for real-time predictions at scale

The problem

Shopify needed a generalized, low-latency online inference solution to serve real-time ML predictions across many internal teams, each with distinct use-case requirements, ML frameworks, and integration needs.

Workflow diagram · grounded in source
1
Merlin Project creation
trigger
“A Merlin Project is a folder in our Merlin mono-repo where the code, configuration and tests of the use case are kept.”
2
Serving layer workspace testing
validation
“our users can create them, run the serving layer in them, and expose a temporary endpoint for development, debugging and stress testing”
3
CI/CD build and Kubernetes deploy
integration
“When a user merges new changes to their repository, Shopify's Buildkite pipeline is automatically triggered and among other actions, builds the image for the service. In the next step of the workflow, that image is then deployed on Shopi…”
4
Feature store access during inference
integration
“Pano, our feature store, can be used to access features in low latency during inference both from the Merlin Online Inference service or from the different clients that send requests to the service.”
5
Real-time model inference
ai_action
“Different clients can call an inference endpoint to generate predictions in real-time. The main clients that use Merlin Online Inference services are Shopify's core services (or any other internal service that requires real-time inference)”
6
Service health monitoring
feedback_loop
“Each service has a monitoring dashboard with predefined metrics such as latency, requests per second, CPU, etc. This can be used to observe the health of the service”
Reported outcome

Merlin Online Inference is in production, empowering data science teams with low latency, scalability, and fast iteration cycles for ML model serving across use cases including fraud detection, product categorization, and inbox classification.

Reported metrics
Inference latencylow latency
Development iteration speedfast iterations
Reported stack
RayComet MLFeastTensorFlowPyTorchXGBoostMLServerFastAPIGoogle Kubernetes EngineFlinkBuildkitePodmanLightGBMScikit-learnHugging Face
Source
https://shopify.engineering/shopifys-machine-learning-platform-real-time-predictions
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Merlin Online Inference is in production, empowering data science teams with low latency, scalability, and fast iteration cycles for ML model serving across use cases including fraud detection, product categorization,…

What tools did this team use?

Ray, Comet ML, Feast, TensorFlow, PyTorch, XGBoost, MLServer, FastAPI, Google Kubernetes Engine, Flink.

What results were reported?

Inference latency: low latency; Development iteration speed: fast iterations (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Merlin Project creation → Serving layer workspace testing → CI/CD build and Kubernetes deploy → Feature store access during inference → Real-time model inference → Service health monitoring.

Related back office ops cases