Back office ops · Production

Salesforce reduces AI inference infrastructure costs up to 8x with Amazon SageMaker AI inference components

The problem

Salesforce's AI Platform team faced two GPU underutilization problems: large models (20–30 GB) with low traffic patterns ran on expensive multi-GPU instances mostly idle, while medium models (~15 GB) handling high-traffic workloads were over-provisioned on similarly expensive multi-GPU setups, both driving avoidable infrastructure cost.

Workflow diagram · grounded in source
1
Identify GPU underutilization
trigger
“The team faced two distinct optimization challenges. Their larger models (20–30 GB) with lower traffic patterns were running on high-performance GPUs, resulting in underutilized multi-GPU instances and inefficient resource allocation. Me…”
2
Configure inference components
integration
“They configured each model, for example, BlockGen and TextEval models as individual inference components specifying precise resource allocations, including accelerator count, memory requirements, model artifacts, container image, and num…”
3
Optimal model packing
ai_action
“SageMaker AI will optimally place and pack models onto ML instances to maximize utilization, leading to cost savings”
4
Independent auto-scaling per model
ai_action
“Each model scales independently based on custom configurations, providing optimal resource allocation to meet specific application requirements”
5
Multi-model GPU sharing
output
“Inference components enable multiple models to share GPU resources efficiently on the same endpoint. This consolidation not only delivers reduction in infrastructure costs through intelligent resource sharing and dynamic scaling”
Reported outcome

By deploying multiple models as inference components on shared SageMaker AI endpoints with dynamic scaling, Salesforce achieved up to an eight-fold reduction in deployment and infrastructure costs while maintaining high performance.

Reported metrics
Deployment and infrastructure cost reductionup to an eight-fold reduction
Infrastructure cost savingssignificant reduction in infrastructure costs
Operational cost reductionsubstantial reduction in operational cost
Reported stack
Amazon SageMaker AIAmazon EC2 P4dCodeGenXGenApexGuru
Source
https://aws.amazon.com/blogs/machine-learning/optimizing-salesforces-model-endpoints-with-amazon-sagemaker-ai-inference-components?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By deploying multiple models as inference components on shared SageMaker AI endpoints with dynamic scaling, Salesforce achieved up to an eight-fold reduction in deployment and infrastructure costs while maintaining hi…

What tools did this team use?

Amazon SageMaker AI, Amazon EC2 P4d, CodeGen, XGen, ApexGuru.

What results were reported?

Deployment and infrastructure cost reduction: up to an eight-fold reduction; Infrastructure cost savings: significant reduction in infrastructure costs; Operational cost reduction: substantial reduction in operational cost (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Identify GPU underutilization → Configure inference components → Optimal model packing → Independent auto-scaling per model → Multi-model GPU sharing.

Related back office ops cases