Ecommerce ops · Production

Shopify fine-tunes a tool-calling agent for Flow: 2.2x faster, 68% cheaper, outperforms closed models

The problem

Store owners who are not engineers found building automation workflows from a blank canvas in Shopify Flow daunting. The feature also faced a cold start problem: no production conversations existed to learn from because Sidekick had not yet been deployed.

First attempt

Offline benchmark results showed parity with the prompt-based agent, but initial production deployment revealed the fine-tuned model had a 35% lower workflow activation rate because synthetic training data did not cover real user requests such as editing existing workflows, handling email configurations, and working with third-party integrations.

Workflow diagram · grounded in source
1
Sample production workflows
trigger
“We reverse-engineered user intent from existing production workflows. Thousands of anonymized store owners had already built workflows manually in Flow. We sampled those and filtered for quality: workflows that had run at least once in t…”
2
Generate synthetic training examples
ai_action
“Generate a user query. Use a stronger LLM to produce a plausible natural-language request that would lead to this workflow. Construct the tool trajectory. Build the full multi-turn sequence of tool calls that an ideal agent would execute…”
3
Translate to Python DSL
ai_action
“switching from the JSON DSL to the Python DSL improved syntactic correctness by 22 points and semantic correctness by 13 points”
4
Fine-tune Qwen3-32B
ai_action
“We fine-tuned Qwen3-32B on this synthetic dataset”
5
Evaluate against benchmark
validation
“evaluated it against a benchmark of 300 hand-crafted examples covering the breadth of expected Flow usage. An LLM evaluation framework compares the generated workflow against the expected one for semantic correctness, and validates synta…”
6
Deploy to production traffic
output
“We deployed it to 1% of traffic to see how it held up”
7
Score, route, and retrain
feedback_loop
“Every production conversation becomes a training signal. We sample high-quality examples: conversations where merchants actually activated the workflow afterwards. The judge scores them, and high-scoring conversations are routed into the…”
Reported outcome

The fine-tuned model is 2.2x faster and 68% cheaper than the closed-model baseline, outperforms closed models, and now serves the majority of production traffic, with a continuous weekly retraining flywheel that closes quality gaps identified in production.

Reported metrics
Inference speed improvement2.2x faster
Inference cost reduction68% cheaper
Workflow activation rate gap at initial deployment vs prompt-based agent35% lower
syntactic correctness improvement (Python DSL vs JSON DSL)22 points
Show all 7 reported metrics
inference speed improvement2.2x faster
inference cost reduction68% cheaper
workflow activation rate gap at initial deployment vs prompt-based agent35% lower
syntactic correctness improvement (Python DSL vs JSON DSL)22 points
semantic correctness improvement (Python DSL vs JSON DSL)13 points
share of failures from email workflows25%
share of failures from diverse condition patterns16%
Reported stack
Qwen3-32BH200 GPUsFSDPTangleCometMLHuggingFaceCentMLSidekick
Source
https://shopify.engineering/fine-tuning-agent-shopify-flow
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The fine-tuned model is 2.2x faster and 68% cheaper than the closed-model baseline, outperforms closed models, and now serves the majority of production traffic, with a continuous weekly retraining flywheel that close…

What tools did this team use?

Qwen3-32B, H200 GPUs, FSDP, Tangle, CometML, HuggingFace, CentML, Sidekick.

What results were reported?

Inference speed improvement: 2.2x faster; Inference cost reduction: 68% cheaper; Workflow activation rate gap at initial deployment vs prompt-based agent: 35% lower; syntactic correctness improvement (Python DSL vs JSON DSL): 22 points (source-reported, not independently verified).

What failed first in this deployment?

Offline benchmark results showed parity with the prompt-based agent, but initial production deployment revealed the fine-tuned model had a 35% lower workflow activation rate because synthetic training data did not cov…

How is this ecommerce ops AI workflow structured?

Sample production workflows → Generate synthetic training examples → Translate to Python DSL → Fine-tune Qwen3-32B → Evaluate against benchmark → Deploy to production traffic → Score, route, and retrain.

Related ecommerce ops cases