Workflow · Production

Beyond accelerators: Lessons from building foundation models on AWS with Japan's GENIAC program

The problem

Allocating over 1,000 accelerators was merely the starting point—successful foundation model training at scale required far more than raw hardware, with the real challenges being reliable distributed systems architecture and cross-organizational coordination.

Workflow diagram · grounded in source
1
GENIAC program launch
trigger
“the Ministry of Economy, Trade and Industry (METI) launched the Generative AI Accelerator Challenge (GENIAC)—a Japanese national program to boost generative AI by providing companies with funding, mentorship, and massive compute resource…”
2
Cross-functional team assembly
integration
“AWS established a virtual team that brought together account teams, specialist Solutions Architects, and service teams”
3
Communication infrastructure setup
integration
“The foundation of our communication strategy was a dedicated internal Slack channel for GENIAC program coordination, connecting AWS account teams with lead SAs. This channel enabled real-time troubleshooting, knowledge sharing, and rapid…”
4
Reference architectures delivered
output
“AWS created pre-validated templates and automation for two main approaches: AWS ParallelCluster (for a user-managed HPC cluster) and SageMaker HyperPod (for a managed, resilient cluster service). These reference architectures covered the…”
5
Mass enablement session
output
“The enablement session welcomed over 80 participants and provided a comprehensive mix of lectures, hands-on labs, and group discussions—earning a CSAT score of 4.75, reflecting its strong impact and relevance to attendees”
6
Customer onboarding with Lead SAs
human_review
“Lead SAs worked directly with teams to deploy training environments, validate setup using NCCL tests, and resolve technical issues in real time”
7
Foundation model training runs
output
“Multiple large language models (LLMs) and custom models were trained successfully, including a 32B multimodal model on Trainium and a 405B tourism-focused multilingual model”
Reported outcome

Twelve customers deployed 127 EC2 P5 instances and 24 EC2 Trn1 instances in a single day, and over 6 months multiple models were trained successfully including a 32B multimodal model on Trainium and a 405B tourism-focused multilingual model.

Reported metrics
Participating customer organizations12
Amazon EC2 P5 instances deployed127
Amazon EC2 Trn1 instances deployed24
Cluster deployment timesingle day
Show all 10 reported metrics
participating customer organizations12
Amazon EC2 P5 instances deployed127
Amazon EC2 Trn1 instances deployed24
cluster deployment timesingle day
program training duration6 months
enablement session participantsover 80
enablement CSAT score4.75
Tokyo event participantsover 50
multimodal model size trained on Trainium32B
multilingual tourism model size405B
Reported stack
Amazon EC2 P5Amazon EC2 Trn1AWS ParallelClusterAmazon EKSAmazon S3Amazon FSx for LustreAmazon FSx for OpenZFSAWS CloudFormationAmazon Managed Service for PrometheusAmazon Managed GrafanaSlurmPyTorchNCCLDCGM ExporterEFA ExporterSlack
Source
https://aws.amazon.com/blogs/machine-learning/beyond-accelerators-lessons-from-building-foundation-models-on-aws-with-japans-geniac-program?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Twelve customers deployed 127 EC2 P5 instances and 24 EC2 Trn1 instances in a single day, and over 6 months multiple models were trained successfully including a 32B multimodal model on Trainium and a 405B tourism-foc…

What tools did this team use?

Amazon EC2 P5, Amazon EC2 Trn1, AWS ParallelCluster, Amazon EKS, Amazon S3, Amazon FSx for Lustre, Amazon FSx for OpenZFS, AWS CloudFormation, Amazon Managed Service for Prometheus, Amazon Managed Grafana.

What results were reported?

Participating customer organizations: 12; Amazon EC2 P5 instances deployed: 127; Amazon EC2 Trn1 instances deployed: 24; Cluster deployment time: single day (source-reported, not independently verified).

How is this workflow AI workflow structured?

GENIAC program launch → Cross-functional team assembly → Communication infrastructure setup → Reference architectures delivered → Mass enablement session → Customer onboarding with Lead SAs → Foundation model training runs.