Order processing · Production

Zalando migrates real-time fraud detection from Python/scikit-learn to Scala/Spark for platform scale

The problem

Zalando's Python-based fraud detection system could not scale to the demands of its expanding fashion platform: the Python GIL blocked multithreading for concurrent predictions, training data exhausted in-house cluster memory, JSON configuration became unmanageable, and shared cluster resources created bottlenecks.

First attempt

The original Python system using CherryPy for serving requests and scikit-learn for ML on a static in-house cluster failed to scale: Python's GIL prevented concurrent predictions, cluster memory capped training data size, and growing JSON config complexity blocked safe refactoring.

Workflow diagram · grounded in source
1
Order received as JSON
trigger
“order data comes in the form of a JSON request”
2
Feature extraction
ai_action
“The request's data fields are read into a data access object, which is used to compute all features and yield a data point”
3
Imputation of missing values
validation
“This initial data point may have missing values since the data may be corrupt or incomplete. Thus, an imputer is used to fill in meaningful default values for them”
4
Fraud probability prediction
ai_action
“The complete data point is now used as input to the final model, which delivers a fraud probability, `P_f`”
5
Order routing by consumers
routing
“This value is then used by the consumers of the service to decide how to proceed with the order”
6
Training data acquisition to S3
integration
“the data is loaded from the sources (databases, logs, etc.), put in a common format, and stored on Amazon's S3 where it is available for learning on AWS”
7
Distributed model training on Spark
ai_action
“the Spark model is learned in a data-distributed setting with one driver and five worker nodes”
Reported outcome

The new Scala and Spark system on AWS reduced overall learning time by a factor of two, cut prediction response time at 20 concurrent requests from ~1000 ms to ~70 ms, and a sparse feature condenser improved prediction accuracy by more than 25% while more than halving runtime.

Reported metrics
Model learning timedrops by a factor of two
Prediction response time at 20 concurrent requests (new vs old)~70 ms vs ~1000 ms
Prediction accuracy improvement from sparse condensermore than 25% improvement
Runtime reduction from sparse condensermore than halved the runtime
Reported stack
scikit-learnCherryPyPlay frameworkMLlibAmazon S3
Source
https://engineering.zalando.com/posts/2016/05/scalable-fraud-detection-fashion-platform.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The new Scala and Spark system on AWS reduced overall learning time by a factor of two, cut prediction response time at 20 concurrent requests from ~1000 ms to ~70 ms, and a sparse feature condenser improved predictio…

What tools did this team use?

scikit-learn, CherryPy, Play framework, MLlib, Amazon S3.

What results were reported?

Model learning time: drops by a factor of two; Prediction response time at 20 concurrent requests (new vs old): ~70 ms vs ~1000 ms; Prediction accuracy improvement from sparse condenser: more than 25% improvement; Runtime reduction from sparse condenser: more than halved the runtime (source-reported, not independently verified).

What failed first in this deployment?

The original Python system using CherryPy for serving requests and scikit-learn for ML on a static in-house cluster failed to scale: Python's GIL prevented concurrent predictions, cluster memory capped training data s…

How is this order processing AI workflow structured?

Order received as JSON → Feature extraction → Imputation of missing values → Fraud probability prediction → Order routing by consumers → Training data acquisition to S3 → Distributed model training on Spark.

Related order processing cases