Quality assurance · Production

Advanced fine-tuning for multi-agent orchestration: production results from Amazon Pharmacy, GES, and A+

The problem

Three Amazon teams faced high-stakes production challenges: Amazon Pharmacy dealt with medication direction errors costing up to $3.5 billion annually; Amazon GES faced lengthy expert-hour inspection reviews for hundreds of fulfillment centers; and Amazon A+ Content needed to evaluate content quality at massive scale across product submissions.

First attempt

Initial attempts using traditional RAG with foundation models at Amazon Pharmacy yielded disappointing results, with accuracy hovering between 60 and 70%, falling short of production requirements.

Workflow diagram · grounded in source
1
High-stakes use case identified
trigger
“One in four high-stakes applications—where patient safety, operational efficiency, or customer trust are on the line—demand advanced fine-tuning and post-training techniques to achieve production-grade performance.”
2
Supervised fine-tuning on domain data
ai_action
“By fine-tuning a model with thousands of expert-annotated examples, Amazon Pharmacy created an agent component that validates medication directions using pharmacy logic and safety guidelines”
3
RLHF/PPO preference refinement
ai_action
“the team further refined the model using PPO incorporating the human feedback data, which boosted the LLM-judge scores from 3.9 to 4.2 out of 5”
4
Group-based reasoning optimization
ai_action
“GRPO generates groups of responses and evaluates each against the average score of the group, rewarding those performing above average while penalizing those below”
5
Domain-specific model evaluation
validation
“evaluation criteria include: drug-drug interaction detection accuracy (percentage of known contraindications correctly identified), dosage calculation precision (correct dosing adjustments for age, weight, and renal function), near-miss …”
6
Deploy as specialized sub-agents
output
“these fine-tuned specialized models will continue to function as domain expert tools within the broader agentic AI system”
7
Continuous monitoring and evaluation
feedback_loop
“it's important to evaluate and monitor your models and agents continuously to ensure high quality and performance”
Reported outcome

Advanced fine-tuning delivered production-grade results across all three Amazon use cases: Amazon Pharmacy achieved a 33% reduction in near-miss medication events; Amazon GES achieved an 80% reduction in human expert effort; and Amazon A+ improved classification accuracy from 77% to 96%.

Reported metrics
medication error reduction (Amazon Pharmacy)33%
human effort reduction (Amazon GES)80%
content quality accuracy improvement (Amazon A+)77% to 96%
RAG accuracy after embedding model fine-tuning (Amazon Pharmacy)90%
Show all 15 reported metrics
medication error reduction (Amazon Pharmacy)33%
human effort reduction (Amazon GES)80%
content quality accuracy improvement (Amazon A+)77% to 96%
RAG accuracy after embedding model fine-tuning (Amazon Pharmacy)90%
customer support contact reduction (Amazon Pharmacy)11%
near-miss event reduction (Amazon Pharmacy)33%
semantic similarity score improvement (Amazon GES)from 0.64 to 0.81
LLM-judge score improvement (Amazon GES)from 3.9 to 4.2 out of 5
domain expert effort reduction (Amazon GES, detail)80%
A+ classification accuracy improvement77% to 96%
annual cost of medication direction errors$3.5 billion annually
production conversion rate with phased maturity approach70–85%
industry average production conversion rate30–40%
year-over-year ROI growth with advanced fine-tuning3-fold year-over-year growth
initial RAG accuracy before fine-tuning (Amazon Pharmacy)60 and 70%
Reported stack
RAGSFTPPODPOGRPODAPOGSPORLHFLoRARLVRRLAIFAmazon SageMakerAmazon BedrockAmazon Bedrock AgentCoreNova LiteAmazon SageMaker HyperPodAmazon SageMaker JumpStartStrandsAmazon Nova Forge
Source
https://aws.amazon.com/blogs/machine-learning/advanced-fine-tuning-techniques-for-multi-agent-orchestration-patterns-from-amazon-at-scale?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Advanced fine-tuning delivered production-grade results across all three Amazon use cases: Amazon Pharmacy achieved a 33% reduction in near-miss medication events; Amazon GES achieved an 80% reduction in human expert…

What tools did this team use?

RAG, SFT, PPO, DPO, GRPO, DAPO, GSPO, RLHF, LoRA, RLVR.

What results were reported?

medication error reduction (Amazon Pharmacy): 33%; human effort reduction (Amazon GES): 80%; content quality accuracy improvement (Amazon A+): 77% to 96%; RAG accuracy after embedding model fine-tuning (Amazon Pharmacy): 90% (source-reported, not independently verified).

What failed first in this deployment?

Initial attempts using traditional RAG with foundation models at Amazon Pharmacy yielded disappointing results, with accuracy hovering between 60 and 70%, falling short of production requirements.

How is this quality assurance AI workflow structured?

High-stakes use case identified → Supervised fine-tuning on domain data → RLHF/PPO preference refinement → Group-based reasoning optimization → Domain-specific model evaluation → Deploy as specialized sub-agents → Continuous monitoring and evaluation.

Related quality assurance cases