Back office ops · Production

Roblox builds hybrid cloud ML inference infrastructure scaling to 250+ AI pipelines in three phases

The problem

The lack of a unified Roblox AI platform caused engineering teams to build fragmented mini-platforms with disparate frameworks, each constructing custom feature engineering, optimizations, and inference scaling solutions independently without central support.

First attempt

The initial offline inference setup was designed only for real-time sequential workloads, lacked support for task parallelism and multistage processing, and required engineers to write their own data chunking and error-handling logic as inference needs scaled.

Workflow diagram · grounded in source
1
Fragmented teams trigger platform need
Trigger
The lack of a unified AI platform led engineering teams to construct their own mini-platforms and select disparate frameworks independently.
▾ source quote
“the lack of a unified Roblox AI platform led engineering teams to construct their own mini platforms and select disparate frameworks”
2
Kubeflow and roblox-ml deployed
Integration
Kubeflow was adopted to provide core ML building blocks including notebooks, pipelines, offline experimentation, and model serving.
▾ source quote
“We adopted Kubeflow early on to take advantage of its packaging of core building blocks for ML, including notebooks, pipelines, offline experimentation, and model serving”
3
Feature store enables feature sharing
Integration
A feature store was built to simplify feature definition and promote sharing of more than 900 features across over 100 feature services.
▾ source quote
“Our feature store simplified the process for defining new features, while promoting the sharing of more than 900 features across over 100 feature services”
4
Extensive pre-release testing
Validation
All models go through offline experiments, shadow testing, and A/B testing before release and are continually monitored for operational performance and accuracy after deployment.
▾ source quote
“models at Roblox go through extensive testing before release. This includes offline experiments, shadow testing, and A/B testing. After being released, models are continually monitored to ensure that they are performing as expected both operationally (e.g., inference latency) and in …”
5
Human moderators review disagreements
Human review
Human moderators evaluate reported disagreements in inferences to ensure correct critical decisions and improve training datasets.
▾ source quote
“human moderators also evaluate any reported disagreements in inferences, which helps ensure that we get critical decisions correct and helps improve the training dataset for our models”
6
Ray enables distributed batch inference
Ai action
Ray was adopted as an open-source compute framework to scale batch inference with multistage processing, task parallelism, and fault tolerance.
▾ source quote
“we added support for Ray, an open-source compute framework that makes it easy to scale batch inference workloads. By building out a Ray-based distributed task pipeline for batch inference, we were able to optimize resource utilization, enable multistage processing, and …”
7
CPU inference moves to own data centers
Integration
All CPU inference was migrated to Roblox's own data centers for direct control over latency and privacy settings.
▾ source quote
“we moved all of our CPU inference to our own data centers, which gave us more direct control over latency and privacy settings”
8
Custom feature store on Feast and Flink
Integration
A custom feature store was built on Feast with Flink as the stream processing engine for real-time features.
▾ source quote
“we developed our own custom feature store, built on top of the open-source project Feast. Our feature store provided a custom domain-specific language for defining transformations for both batch and streaming features. Flink was adopted as the stream processing engine …”
9
ML gateway centralizes model access
Integration
A unified ML gateway was built to centralize access to all large models with a common API interface and efficient resource sharing.
▾ source quote
“We built our unified ML gateway to centralize access to all large models, both open source and internally developed, across a variety of environments, including CPUs and GPUs in the cloud and on premises. Our goal was to create an …”
10
vLLM adopted as LLM inference engine
Ai action
vLLM was adopted as the primary LLM inference engine, yielding an almost 2x improvement in both latency and throughput.
▾ source quote
“we have adopted vLLM as our primary inference engine for LLMs, leveraging vLLM's high-performance capabilities to power AI applications across Roblox. Since moving to vLLM, we've seen an almost 2x improvement in both latency and throughput, and we currently serve …”
Reported outcome

Roblox scaled from fewer than 50 ML inference pipelines to approximately 250, with vLLM delivering an almost 2x improvement in latency and throughput and currently serving approximately 4 billion tokens per week; Avatar Auto Setup produced approximately 8% of UGC avatar bodies as of August 2024.

Reported metrics
ML inference pipelines (early 2023)fewer than 50
ML inference pipelines (current)approximately 250
UGC avatar bodies produced using Avatar Auto Setupapproximately 8%
tokens processed per week (Roblox Assistant)1.5 billion per week
Show all 14 reported metrics
ML inference pipelines (early 2023)fewer than 50
ML inference pipelines (current)approximately 250
UGC avatar bodies produced using Avatar Auto Setupapproximately 8%
tokens processed per week (Roblox Assistant)1.5 billion per week
latency and throughput improvement with vLLMalmost 2x improvement
tokens served per week (vLLM)approximately 4 billion tokens per week
daily personalization requestsapproximately 1 billion
daily active users79.5 million
feature store records ingested per dayapproximately 30 billion
feature store records served per dayapproximately 70 billion
feature store P99 latency50ms
features shared across feature servicesmore than 900
CLIP embedding compute costsignificantly reduced
parallel A/B testsapproximately 20
Reported stack
KubeflowJupyterKServeTriton Inference ServerRayFeastFlinkvLLMvector databaseCLIPRoblox Assistant
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://corp.roblox.com/newsroom/2024/09/running-ai-inference-at-scale-in-the-hybrid-cloud
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Roblox scaled from fewer than 50 ML inference pipelines to approximately 250, with vLLM delivering an almost 2x improvement in latency and throughput and currently serving approximately 4 billion tokens per week; Avat…

What tools did this team use?

Kubeflow, Jupyter, KServe, Triton Inference Server, Ray, Feast, Flink, vLLM, vector database, CLIP.

What results were reported?

ML inference pipelines (early 2023): fewer than 50; ML inference pipelines (current): approximately 250; UGC avatar bodies produced using Avatar Auto Setup: approximately 8%; tokens processed per week (Roblox Assistant): 1.5 billion per week (source-reported, not independently verified).

What failed first in this deployment?

The initial offline inference setup was designed only for real-time sequential workloads, lacked support for task parallelism and multistage processing, and required engineers to write their own data chunking and erro…

How is this back office ops AI workflow structured?

Fragmented teams trigger platform need → Kubeflow and roblox-ml deployed → Feature store enables feature sharing → Extensive pre-release testing → Human moderators review disagreements → Ray enables distributed batch inference → CPU inference moves to own data centers → Custom feature store on Feast and Flink → ML gateway centralizes model access → vLLM adopted as LLM inference engine.

Related back office ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Back office ops cases →