Ecommerce ops · Production

Zalando's Machine Learning Platform: from experimentation notebooks to production pipelines at scale

The problem

Crossing the gap between notebook-based ML experimentation and production-grade pipelines was the core challenge: Jupyter notebooks do not scale to production requirements (security, reproducibility, observability, performance), and manually writing CloudFormation templates for pipelines was verbose and error-prone.

First attempt

CloudFormation templates for Step Functions pipelines became too verbose and tedious to edit manually at scale, requiring an internal abstraction layer (zflow) to remain maintainable.

Workflow diagram · grounded in source
1
Explore data in Datalab
trigger
“Zalando provides its ML practitioners with access to a hosted version of JupyterHub, an experimentation platform where they can use Jupyter notebooks, R Studio, and other tools they may need to query available data, visualize results, an…”
2
Scale up with Databricks/Spark
ai_action
“Apache Spark is much better suited for that purpose, and Zalando users can access it via Databricks. It's a well-known tool within the data science community, suitable for both experimentation via notebooks and for running large-scale da…”
3
Define pipeline with zflow DSL
integration
“A pipeline in a zflow script is a Python object with a series of stages attached to it. zflow provides a number of custom functions for configuring ML tasks, for example training, batch transform, and hyperparameter tuning. It also offer…”
4
Generate CloudFormation template
integration
“When a pipeline script is executed, zflow uses AWS CDK to generate a CloudFormation template file. The file contains all the information needed to create the necessary AWS resources.”
5
Deploy via Zalando CDP
integration
“All that is needed now is to commit and push the generated template to the git repository and let Zalando Continuous Delivery Platform (CDP) deploy it to AWS.”
6
Execute Step Functions pipeline
ai_action
“our pipeline will appear in the AWS Console as a Step Functions state machine. It can then be executed, either via scheduler (like in our example), manually in the Console, or programatically via an API call.”
7
Monitor via ML portal
feedback_loop
“Pipeline authors can monitor how metrics evolve across multiple runs of training pipelines and can view these changes on a graph. They can also view model cards for models created by the pipelines.”
Reported outcome

zflow has been used to create hundreds of ML pipelines at Zalando, and the tooling abstracts away infrastructure complexity so ML practitioners can focus on their domain rather than the infrastructure.

Reported metrics
ML pipelines created with zflowhundreds of pipelines
time to start experimenting in Datalabless than a minute
Reported stack
JupyterHubDatabricksApache SparkAmazon SageMakerAWS Step FunctionsAWS CDKCloudFormationAWS LambdaS3BigQueryMicroStrategyBackstagezflowHPC
Source
https://engineering.zalando.com/posts/2022/04/zalando-machine-learning-platform.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

zflow has been used to create hundreds of ML pipelines at Zalando, and the tooling abstracts away infrastructure complexity so ML practitioners can focus on their domain rather than the infrastructure.

What tools did this team use?

JupyterHub, Databricks, Apache Spark, Amazon SageMaker, AWS Step Functions, AWS CDK, CloudFormation, AWS Lambda, S3, BigQuery.

What results were reported?

ML pipelines created with zflow: hundreds of pipelines; time to start experimenting in Datalab: less than a minute (source-reported, not independently verified).

What failed first in this deployment?

CloudFormation templates for Step Functions pipelines became too verbose and tedious to edit manually at scale, requiring an internal abstraction layer (zflow) to remain maintainable.

How is this ecommerce ops AI workflow structured?

Explore data in Datalab → Scale up with Databricks/Spark → Define pipeline with zflow DSL → Generate CloudFormation template → Deploy via Zalando CDP → Execute Step Functions pipeline → Monitor via ML portal.

Related ecommerce ops cases