Ecommerce ops · Production

Shop the Look: Zalando's deep learning visual search pipeline for fashion product retrieval

The problem

Users want to find fashion products they see in photos, but words alone are insufficient to describe fashion items. Visual search for fashion poses challenges around image quality, lighting, varied backgrounds, human poses, and article distortion at scale.

First attempt

Studio2Shop was trained exclusively on clean-background studio images, making it unsuitable for natural real-world photos. A direct extension to natural images was blocked by the absence of annotated natural fashion image data.

Workflow diagram · grounded in source
1
User submits query image
trigger
“At Zalando, we have many outlets where this search is possible: our app, our Facebook chatbot, etc.”
2
Background segmentation
ai_action
“Street2Fashion, a U-net-like segmentation model that can find the person in the image and simply replaces the background with white pixels”
3
Product feature extraction
ai_action
“FashionDNA is run on the title images of the products in the assortment (bottom row) to obtain static feature vectors”
4
Product matching and ranking
ai_action
“a query fashion image is processed by the segmentation model to remove the background, and can then go through the matching model described above to be matched with appropriate products”
5
Top product suggestions output
output
“the top 50 product suggestions”
Reported outcome

Zalando developed the Street2Fashion2Shop pipeline combining background segmentation and product matching to handle real-world query images.
The segmentation results were described as good enough to focus on the fashion in the image. The work remained at research stage at time of publication; production visual search was powered by Fashwell.

Reported metrics
Segmentation quality for natural imagesgood enough to focus on the fashion in the image
Reported stack
FashionDNAStudio2ShopStreet2FashionFashion2Shop
Source
https://engineering.zalando.com/posts/2018/09/shop-look-deep-learning.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Zalando developed the Street2Fashion2Shop pipeline combining background segmentation and product matching to handle real-world query images.

What tools did this team use?

FashionDNA, Studio2Shop, Street2Fashion, Fashion2Shop.

What results were reported?

Segmentation quality for natural images: good enough to focus on the fashion in the image (source-reported, not independently verified).

What failed first in this deployment?

Studio2Shop was trained exclusively on clean-background studio images, making it unsuitable for natural real-world photos.

How is this ecommerce ops AI workflow structured?

User submits query image → Background segmentation → Product feature extraction → Product matching and ranking → Top product suggestions output.

Related ecommerce ops cases