Ecommerce ops · Production

Shopify builds production-ready agentic systems for Sidekick with JIT instructions, LLM evaluation, and GRPO training

The problem

As Sidekick's tool inventory grew beyond 50 specialized capabilities, the system prompt became an unwieldy collection of special cases nearly impossible to maintain, and traditional software testing approaches fell short for evaluating the probabilistic, multi-step nature of LLM-based agents.

First attempt

Vibe-testing with simple 0-to-10 LLM judges yielded near-random evaluation quality (Cohen's Kappa of 0.02), and GRPO training produced significant reward hacking — opt-out behavior, tag misuse, and schema violations — that undermined model improvements.

Workflow diagram · grounded in source
1
Merchant natural language request
trigger
“a human provides input, an LLM processes that input and decides on actions”
2
JIT instructions served to LLM
ai_action
“we return relevant instructions alongside tool data exactly when they're needed. Our goal is to craft the perfect context for the LLM for every single situation, not a token less, not a token more”
3
LLM acts in agentic loop
ai_action
“an LLM processes that input and decides on actions, those actions are executed in the environment, feedback is collected, and the cycle continues until the task is complete”
4
LLM judges evaluate outputs
validation
“improved our judges from barely-better-than-random (Cohen's Kappa of 0.02) to near-human performance (0.61 vs. human baseline of 0.69)”
5
Merchant simulator pre-production test
validation
“we built an LLM-powered merchant simulator that captures the "essence" or goals of real conversations and replays them through new system candidates”
6
GRPO training with gated rewards
feedback_loop
“we implemented Group Relative Policy Optimization (GRPO), a reinforcement learning approach that uses our LLM judges as reward signals. We developed an N-Stage Gated Rewards system that combines procedural validation (syntax checking, sc…”
Reported outcome

After implementing JIT instructions, improved LLM judges, and reward-hacking fixes, syntax validation accuracy improved from ~93% to ~99%, LLM judge correlation improved from 0.66 to 0.75 on average, and end-to-end conversation quality matched the supervised fine-tuning baseline.

Reported metrics
Syntax validation accuracy after fix~99%
Syntax validation accuracy before fix~93%
LLM judge correlation after improvement0.75
LLM judge correlation before improvement0.66
Show all 8 reported metrics
syntax validation accuracy after fix~99%
syntax validation accuracy before fix~93%
LLM judge correlation after improvement0.75
LLM judge correlation before improvement0.66
LLM judge Cohen's Kappa initial0.02
LLM judge Cohen's Kappa improved0.61
human evaluator baseline Cohen's Kappa0.69
overall system performance after JITperformance improved across all metrics
Reported stack
SidekickGRPO
Source
https://shopify.engineering/building-production-ready-agentic-systems?utm_source=substack&utm_medium=email
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After implementing JIT instructions, improved LLM judges, and reward-hacking fixes, syntax validation accuracy improved from ~93% to ~99%, LLM judge correlation improved from 0.66 to 0.75 on average, and end-to-end co…

What tools did this team use?

Sidekick, GRPO.

What results were reported?

Syntax validation accuracy after fix: ~99%; Syntax validation accuracy before fix: ~93%; LLM judge correlation after improvement: 0.75; LLM judge correlation before improvement: 0.66 (source-reported, not independently verified).

What failed first in this deployment?

Vibe-testing with simple 0-to-10 LLM judges yielded near-random evaluation quality (Cohen's Kappa of 0.02), and GRPO training produced significant reward hacking — opt-out behavior, tag misuse, and schema violations —…

How is this ecommerce ops AI workflow structured?

Merchant natural language request → JIT instructions served to LLM → LLM acts in agentic loop → LLM judges evaluate outputs → Merchant simulator pre-production test → GRPO training with gated rewards.

Related ecommerce ops cases