Finance ops · Production

Zalando rebuilds ML pipeline for payment-default fraud detection on Amazon SageMaker

The problem

Zalando's second-generation Scala/Spark ML pipeline for detecting payment defaults was tightly coupled to a single framework making modern Python libraries difficult to adopt, relied on custom code that added maintenance burden, suffered from memory issues and latency spikes with slow instance startup, and had a monolithic design that fused feature preprocessing with model training into a single cluster.

First attempt

An original Python/scikit-learn ML setup was replaced in 2015 by a Scala/Spark system to scale better, but this second-generation system accumulated its own technical pain points that necessitated a third migration.

Workflow diagram · grounded in source
1
Training data preprocessing
ai_action
“Training data preprocessing, using a Databricks cluster and a scikit-learn batch transform job on SageMaker”
2
Model training
ai_action
“Training a model using a SageMaker training job”
3
Batch prediction generation
ai_action
“Generating predictions with another batch transform job”
4
Model evaluation report
validation
“we added a custom Databricks job to calculate those metrics and to plot them in a PDF report”
5
SageMaker endpoint deployment
output
“Deploying a SageMaker endpoint to serve the model”
6
Order request received
trigger
“We receive a JSON response with order data, and return a response in a JSON format”
7
Feature extraction
ai_action
“A scikit-learn container for processing the incoming requests, i.e. extracting features from the input JSON or basic data transformations”
8
Default probability prediction
ai_action
“Main model container (i.e. XGBoost, PyTorch) for model predictions”
Reported outcome

The new SageMaker-based pipeline is framework-independent with clear separation between preprocessing and training, and reduced scale-up time by 50%.
Load tests confirm a single ml.m5.large instance handles 200 requests/second with p99 latency under 80ms.

Reported metrics
Scale-up time reduction50%
p99 latency at 200 RPS (single ml.m5.large)under 80ms
response times at 1000 RPSbelow 200ms
Serving cost increase vs legacyup to 200%
Show all 5 reported metrics
scale-up time reduction50%
p99 latency at 200 RPS (single ml.m5.large)under 80ms
response times at 1000 RPSbelow 200ms
serving cost increase vs legacyup to 200%
latency SLA compliance requirement99.9%
Reported stack
Amazon SageMakerzflowAWS Step FunctionsAWS LambdasDatabricksscikit-learnXGBoostPyTorchTensorflow
Source
https://engineering.zalando.com/posts/2021/02/machine-learning-pipeline-with-real-time-inference.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The new SageMaker-based pipeline is framework-independent with clear separation between preprocessing and training, and reduced scale-up time by 50%.

What tools did this team use?

Amazon SageMaker, zflow, AWS Step Functions, AWS Lambdas, Databricks, scikit-learn, XGBoost, PyTorch, Tensorflow.

What results were reported?

Scale-up time reduction: 50%; p99 latency at 200 RPS (single ml.m5.large): under 80ms; response times at 1000 RPS: below 200ms; Serving cost increase vs legacy: up to 200% (source-reported, not independently verified).

What failed first in this deployment?

An original Python/scikit-learn ML setup was replaced in 2015 by a Scala/Spark system to scale better, but this second-generation system accumulated its own technical pain points that necessitated a third migration.

How is this finance ops AI workflow structured?

Training data preprocessing → Model training → Batch prediction generation → Model evaluation report → SageMaker endpoint deployment → Order request received → Feature extraction → Default probability prediction.

Related finance ops cases