Back office ops · Production

HubX achieves 2.5x faster inference and 40% cost reduction with Google Kubernetes Engine and Trillium TPUs

The problem

HubX required AI-powered mobile apps to deliver responses within 10 seconds to prevent user churn, but its previous infrastructure produced slow processing times and latency issues, making rapid iteration and deployment difficult.

First attempt

HubX's previous infrastructure caused slow processing times and latency issues that drove high user churn and blocked rapid iteration.

Workflow diagram · grounded in source
1
Serverless app deployment
integration
“The company also uses Cloud Run and Cloud Run functions for rapid, serverless deployment of AI/ML applications”
2
Migration to GKE at scale
integration
“then migrates them to GKE for better management and control as they scale”
3
Hardware-optimized model inference
ai_action
“using TPUs for fine-tuning and inference, A100 GPUs for larger, more complex models, and L4 GPUs for workloads that require fewer resources to optimize costs”
4
High-performance model storage
integration
“With Hyperdisk ML, we've unlocked impressive 20-30x faster boot-up times, ensuring that even the most complex ML workloads are ready to serve in record time”
5
Sub-10-second user response
output
“Users now get a response to queries in less than 10 seconds, improving user experience, conversion rates, engagement, and retention”
Reported outcome

After adopting GKE, HubX achieved 2.5x faster inference speeds via Trillium TPUs, reduced operating costs by 40%, delivered user query responses in under 10 seconds, and unlocked 20-30x faster model boot-up times with cold start times reduced by one full minute.

Reported metrics
inference speed improvement (Trillium TPUs)2.5x faster
Operating cost reduction40%
User query response timeless than 10 seconds
model boot-up time improvement (Hyperdisk ML)20-30x faster
Show all 7 reported metrics
inference speed improvement (Trillium TPUs)2.5x faster
operating cost reduction40%
user query response timeless than 10 seconds
model boot-up time improvement (Hyperdisk ML)20-30x faster
cold start time reductionone full minute
inference speed improvement (GKE additional)10%
user churn threshold (prompt-to-output wait)30 seconds
Reported stack
Google Kubernetes EngineAI HypercomputerTrillium TPUsA100 GPUsL4 GPUsCloud RunCloud Run functionsHyperdisk MLGoogle Cloud
Source
https://cloud.google.com/customers/hubx-ai
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After adopting GKE, HubX achieved 2.5x faster inference speeds via Trillium TPUs, reduced operating costs by 40%, delivered user query responses in under 10 seconds, and unlocked 20-30x faster model boot-up times with…

What tools did this team use?

Google Kubernetes Engine, AI Hypercomputer, Trillium TPUs, A100 GPUs, L4 GPUs, Cloud Run, Cloud Run functions, Hyperdisk ML, Google Cloud.

What results were reported?

inference speed improvement (Trillium TPUs): 2.5x faster; Operating cost reduction: 40%; User query response time: less than 10 seconds; model boot-up time improvement (Hyperdisk ML): 20-30x faster (source-reported, not independently verified).

What failed first in this deployment?

HubX's previous infrastructure caused slow processing times and latency issues that drove high user churn and blocked rapid iteration.

How is this back office ops AI workflow structured?

Serverless app deployment → Migration to GKE at scale → Hardware-optimized model inference → High-performance model storage → Sub-10-second user response.

Related back office ops cases