Workflow ·

Journey to 3200 Gbps: High-Performance GPU Memory Transfer on AWS SageMaker HyperPod

The problem

Perplexity needed to efficiently transfer non-contiguous GPU memory regions between machines at maximum possible speed on AWS p5 instances, while supporting dynamic node addition and removal without disrupting operations. NCCL, the de facto standard library, was unsuitable because it requires a static cluster world and uses a synchronous communication model incompatible with their asynchronous workload.

First attempt

NCCL was not ideal: it requires a static cluster world causing a full cluster restart when nodes change, its synchronous model added complexity for an asynchronous workload, and it did not permit direct control over memory transfer patterns.

Workflow diagram · grounded in source
1
Basic SEND/RECV transfer
trigger
“Implemented basic unidirectional message transfer using SEND/RECV”
2
Bidirectional communication
integration
“Extended to bidirectional communication”
3
GPUDirect RDMA WRITE
integration
“Added GPUDirect RDMA WRITE for direct GPU-GPU transfer”
4
Multi-card scaling optimizations
output
“When scaling to 32 network cards, we implemented several crucial optimizations”
5
Peak bandwidth achieved
output
“we achieved a final performance of 3,108 Gbps - 97.1% of the theoretical maximum bandwidth”
Reported outcome

Perplexity's custom libfabric-based RDMA solution achieved 3,108 Gbps — 97.1% of the theoretical 3,200 Gbps maximum — across all network cards on AWS p5 instances.

Reported metrics
Final bandwidth achieved3,108 Gbps
Percentage of theoretical maximum bandwidth97.1%
Theoretical maximum network bandwidth3200 Gbps
Bandwidth utilization on single network card97.4%
Reported stack
libfabricEFARDMAGPUDirect RDMANCCLKubernetes
Source
https://www.perplexity.ai/hub/blog/high-performance-gpu-memory-transfer-on-aws
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Perplexity's custom libfabric-based RDMA solution achieved 3,108 Gbps — 97.1% of the theoretical 3,200 Gbps maximum — across all network cards on AWS p5 instances.

What tools did this team use?

libfabric, EFA, RDMA, GPUDirect RDMA, NCCL, Kubernetes.

What results were reported?

Final bandwidth achieved: 3,108 Gbps; Percentage of theoretical maximum bandwidth: 97.1%; Theoretical maximum network bandwidth: 3200 Gbps; Bandwidth utilization on single network card: 97.4% (source-reported, not independently verified).

What failed first in this deployment?

NCCL was not ideal: it requires a static cluster world causing a full cluster restart when nodes change, its synchronous model added complexity for an asynchronous workload, and it did not permit direct control over m…

How is this workflow AI workflow structured?

Basic SEND/RECV transfer → Bidirectional communication → GPUDirect RDMA WRITE → Multi-card scaling optimizations → Peak bandwidth achieved.