Ecommerce ops · Production

Shopify builds Vision Language Model product classification system processing 30M daily predictions

The problem

Shopify's early product classification systems struggled with increasing product complexity and diversity on the platform. By early 2023, the team identified unmet requirements including more granular product understanding, consistent taxonomy, attribute extraction, richer metadata, and content safety features that category-only classification could not satisfy.

First attempt

The initial 2018 logistic regression with TF-IDF classifier was effective only for simple cases, and the 2020 multi-modal system still fell short because classifying categories alone was insufficient for comprehensive product understanding.

Workflow diagram · grounded in source
1
Dynamic product request batching
trigger
“Instead of pre-defining fixed batch sizes, our system dynamically groups incoming product requests based on real-time arrival patterns.”
2
Input validation
validation
“Preliminary validation of product data”
3
VLM category prediction
ai_action
“Category prediction with simplified description generation”
4
VLM attribute prediction
ai_action
“The prompt of the second call is dependent on the output of the first call, as the attributes we predict for a product depend on the category of the product.”
5
Consistency check and retry
validation
“Both category and attribute predictions must succeed”
6
Multi-LLM training annotation
ai_action
“our multi-LLM annotation system, where several large language models independently evaluate each product”
7
Human review of edge cases
human_review
“we incorporate a human validation layer focused on strategic manual review of complex edge cases and novel product types”
8
Output storage and notification
output
“Validation against taxonomy rules”
9
Continuous quality feedback
feedback_loop
“This introduces a continuous feedback loop for ongoing improvement and adaptability. Regular quality audits are conducted as part of this process”
Reported outcome

The VLM-based system achieves an 85% merchant acceptance rate for predicted categories, processes over 30 million predictions daily, and has doubled hierarchical precision and recall compared to the earlier neural network approach.

Reported metrics
Merchant acceptance rate of predicted categories85%
Daily predictions processedover 30 million predictions daily
Hierarchical precision and recall vs earlier neural networkdoubled compared to our earlier neural network approach
Historical products in systembillions of historical products
Reported stack
Vision Language ModelsLlaVA 1.5 7BLLaMA 3.2 11BQwen2VL 7BNvidia DynamoKubernetesDataflow
Source
https://shopify.engineering/evolution-product-classification
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The VLM-based system achieves an 85% merchant acceptance rate for predicted categories, processes over 30 million predictions daily, and has doubled hierarchical precision and recall compared to the earlier neural net…

What tools did this team use?

Vision Language Models, LlaVA 1.5 7B, LLaMA 3.2 11B, Qwen2VL 7B, Nvidia Dynamo, Kubernetes, Dataflow.

What results were reported?

Merchant acceptance rate of predicted categories: 85%; Daily predictions processed: over 30 million predictions daily; Hierarchical precision and recall vs earlier neural network: doubled compared to our earlier neural network approach; Historical products in system: billions of historical products (source-reported, not independently verified).

What failed first in this deployment?

The initial 2018 logistic regression with TF-IDF classifier was effective only for simple cases, and the 2020 multi-modal system still fell short because classifying categories alone was insufficient for comprehensive…

How is this ecommerce ops AI workflow structured?

Dynamic product request batching → Input validation → VLM category prediction → VLM attribute prediction → Consistency check and retry → Multi-LLM training annotation → Human review of edge cases → Output storage and notification → Continuous quality feedback.

Related ecommerce ops cases