Ecommerce ops · Production

How Amazon scaled Rufus by building multi-node inference using AWS Trainium chips and vLLM

The problem

As the Rufus LLM grew larger, no single accelerator or instance had enough memory for the entire model, requiring Amazon to engineer scalable multi-node inference that could maintain low latency and cost-efficiency while managing distributed model sharding and inter-node coordination.

Workflow diagram · grounded in source
1
Customer shopping query arrives
trigger
“a generative AI-powered shopping assistant that serves millions of customers at immense scale”
2
Proxy routes and monitors traffic
routing
“a proxy layer positioned between the system's ingress or load-balancing layer and the multi-node inference unit. This proxy layer is responsible for continuously probing and reporting the health of all worker nodes”
3
Leader node orchestrates inference
ai_action
“the leader node runs the Triton Inference Server and vLLM engine, serving as the primary orchestration unit for inference”
4
Model inputs broadcast to followers
integration
“Model inputs are broadcasted from the leader node to follower nodes using the torch.distributed communication library with the Gloo backend”
5
Parallel distributed model execution
ai_action
“continuously listens to model inputs broadcasted from the leader node and executes the model using the Neuron runtime in parallel with the leader node”
6
Response delivered at scale
output
“delivering a highly available, high-throughput, multi-node LLM inference solution at industry scale”
Reported outcome

Amazon successfully launched a much larger Rufus model across tens of thousands of Trainium chips, supporting Prime Day traffic, with the increased model capacity enabling new shopping experiences and significantly improved user engagement.

Reported metrics
Trainium chips deployedover tens of thousands
User engagementsignificantly improved user engagement
Overall throughputsignificantly improving overall throughput
Solution availabilityhighly available
Reported stack
Amazon TrainiumvLLMAmazon ECSNVIDIA Triton Inference ServerNeuron SDKEFANeuronWorker
Source
https://aws.amazon.com/blogs/machine-learning/how-amazon-scaled-rufus-by-building-multi-node-inference-using-aws-trainium-chips-and-vllm?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Amazon successfully launched a much larger Rufus model across tens of thousands of Trainium chips, supporting Prime Day traffic, with the increased model capacity enabling new shopping experiences and significantly im…

What tools did this team use?

Amazon Trainium, vLLM, Amazon ECS, NVIDIA Triton Inference Server, Neuron SDK, EFA, NeuronWorker.

What results were reported?

Trainium chips deployed: over tens of thousands; User engagement: significantly improved user engagement; Overall throughput: significantly improving overall throughput; Solution availability: highly available (source-reported, not independently verified).

How is this ecommerce ops AI workflow structured?

Customer shopping query arrives → Proxy routes and monitors traffic → Leader node orchestrates inference → Model inputs broadcast to followers → Parallel distributed model execution → Response delivered at scale.

Related ecommerce ops cases