Quality assurance · Production

Training Llama 3.3 Swallow: A Japanese Sovereign LLM on Amazon SageMaker HyperPod

The problem

The Institute of Science Tokyo sought to build a large language model with enhanced Japanese capabilities capable of surpassing existing leading models, requiring efficient large-scale distributed training infrastructure on cloud.

Workflow diagram · grounded in source
1
Corpus quality filtering
ai_action
“the team employed the Swallow Education Classifier to extract educationally valuable content from the corpus”
2
Data preload to HPC storage
integration
“training data was preloaded from Amazon S3 to the FSx for Lustre file system to prevent I/O bottlenecks during training”
3
Memory prediction and config optimization
validation
“This tool accurately predicts per-GPU memory usage during training and semi-automatically determines optimal training settings by analyzing all possible 4D parallelism configurations”
4
Distributed continual pre-training
ai_action
“The team used a comprehensive 4D parallelism strategy of Megatron-LM that maximizes GPU utilization through careful optimization of communication patterns across multiple dimensions: data, tensor, and pipeline, and sequence parallelism”
5
Supervised instruction fine-tuning
ai_action
“the team focused exclusively on Japanese dialogue and code generation tasks. This version was created through supervised fine-tuning of the base model”
6
Benchmarking against leading models
validation
“In comprehensive evaluations, it has shown superior capabilities compared to OpenAI's GPT-4o (gpt-4o-2024-08-06), GPT-4o-mini (gpt-4o-mini-2024-07-18), GPT-3.5 (gpt-3.5-turbo-0125), and Qwen2.5-72B”
7
Automated training monitoring
feedback_loop
“By integrating with Weights & Biases, the system continuously monitors training progress and delivers automated notifications for key events such as job completion or failure and performance anomalies. Weights & Biases provides a set of …”
Reported outcome

Llama 3.3 Swallow outperforms GPT-4o, GPT-4o-mini, GPT-3.5, and Qwen2.5-72B on Japanese benchmarks, and the distributed checkpointing system saves checkpoints up to 10 times faster compared to synchronous approaches.

Reported metrics
Checkpoint save speed improvement10 times faster
Training data tokensapproximately 314 billion tokens
Continual pre-training duration16 days and 6 hours
GPU count for training256 GPUs
Reported stack
Amazon SageMaker HyperPodAmazon EC2Amazon S3Amazon FSx for LustreMegatron-LMWeights & BiasesNCCLAWS-OFI-NCCLAmazon Managed Service for PrometheusAmazon Managed GrafanaDCGM ExporterEFA ExporterElastic Fabric Adapter (EFA)SlurmAWS CloudFormationSwallow Education ClassifierHugging FaceSlack
Source
https://aws.amazon.com/blogs/machine-learning/training-llama-3-3-swallow-a-japanese-sovereign-llm-on-amazon-sagemaker-hyperpod?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Llama 3.3 Swallow outperforms GPT-4o, GPT-4o-mini, GPT-3.5, and Qwen2.5-72B on Japanese benchmarks, and the distributed checkpointing system saves checkpoints up to 10 times faster compared to synchronous approaches.

What tools did this team use?

Amazon SageMaker HyperPod, Amazon EC2, Amazon S3, Amazon FSx for Lustre, Megatron-LM, Weights & Biases, NCCL, AWS-OFI-NCCL, Amazon Managed Service for Prometheus, Amazon Managed Grafana.

What results were reported?

Checkpoint save speed improvement: 10 times faster; Training data tokens: approximately 314 billion tokens; Continual pre-training duration: 16 days and 6 hours; GPU count for training: 256 GPUs (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Corpus quality filtering → Data preload to HPC storage → Memory prediction and config optimization → Distributed continual pre-training → Supervised instruction fine-tuning → Benchmarking against leading models → Automated training monitoring.

Related quality assurance cases