Ecommerce ops · Production

Shopify builds real-time ML embedding pipelines processing 2,500 embeddings per second for Semantic Search

The problem

Shopify merchants needed search results that reflected consumer intent beyond keyword matching, and product and image updates needed to be reflected in search results nearly instantly after creation or modification.

First attempt

The initial image embedding pipeline ran into Out of Memory errors on n1-standard-16 machines; switching to n1-highmem-16 machines resolved the OOM issue but increased costs by 14%. Batching also proved ineffective — due to bursty input topics, elements were being organized in bundles of 1, sending batches of 1 to the GPU.

Workflow diagram · grounded in source

Event triggers pipeline

trigger

“The pipeline listens to new events from an input event topic that signals an image has been created or modified on a merchant's website”

Image preprocessing

integration

“The new event is then preprocessed before running inference: Download the image to the worker machine Load the image to memory Resize the image”

Embedding vector generation

ai_action

“Next, the embedding vector is generated for the image. Some final postprocessing is applied to the embeddings.”

Write to data warehouse and event topic

output

“The embedding is written to: Data warehouse for offline analysis (reports, dashboards) An output event topic for downstream real time ingestion (Shopify Storefront Search)”

Reported outcome

Shopify now processes roughly 2,500 embeddings per second (216 million per day) in near real time across image and text pipelines, achieved a ~2.6x memory footprint decrease, and eliminated the extra 14% in cost by reverting to n1-standard-16 machines.

Reported metrics

Embeddings processed per second2,500

Embeddings processed per day216 million

Memory footprint decrease~2.6x

Worker cost increase (eliminated by optimization)14%

Reported stack

DataflowBigQueryApache BeamT4 GPU

Source

https://shopify.engineering/how-shopify-improved-consumer-search-intent-with-real-time-ml

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Shopify now processes roughly 2,500 embeddings per second (216 million per day) in near real time across image and text pipelines, achieved a ~2.6x memory footprint decrease, and eliminated the extra 14% in cost by re…

What tools did this team use?

Dataflow, BigQuery, Apache Beam, T4 GPU.

What results were reported?

Embeddings processed per second: 2,500; Embeddings processed per day: 216 million; Memory footprint decrease: ~2.6x; Worker cost increase (eliminated by optimization): 14% (source-reported, not independently verified).

What failed first in this deployment?

The initial image embedding pipeline ran into Out of Memory errors on n1-standard-16 machines; switching to n1-highmem-16 machines resolved the OOM issue but increased costs by 14%.

How is this ecommerce ops AI workflow structured?

Event triggers pipeline → Image preprocessing → Embedding vector generation → Write to data warehouse and event topic.