Back office ops · Production

Roblox builds hybrid cloud ML inference infrastructure scaling to 250+ AI pipelines in three phases

The problem

The lack of a unified Roblox AI platform caused engineering teams to build fragmented mini-platforms with disparate frameworks, each constructing custom feature engineering, optimizations, and inference scaling solutions independently without central support.

First attempt

The initial offline inference setup was designed only for real-time sequential workloads, lacked support for task parallelism and multistage processing, and required engineers to write their own data chunking and error-handling logic as inference needs scaled.

Workflow diagram · grounded in source
1
Fragmented teams trigger platform need
trigger
“the lack of a unified Roblox AI platform led engineering teams to construct their own mini platforms and select disparate frameworks”
2
Kubeflow and roblox-ml deployed
integration
“We adopted Kubeflow early on to take advantage of its packaging of core building blocks for ML, including notebooks, pipelines, offline experimentation, and model serving”
3
Feature store enables feature sharing
integration
“Our feature store simplified the process for defining new features, while promoting the sharing of more than 900 features across over 100 feature services”
4
Extensive pre-release testing
validation
“models at Roblox go through extensive testing before release. This includes offline experiments, shadow testing, and A/B testing. After being released, models are continually monitored to ensure that they are performing as expected both …”
5
Human moderators review disagreements
human_review
“human moderators also evaluate any reported disagreements in inferences, which helps ensure that we get critical decisions correct and helps improve the training dataset for our models”
6
Ray enables distributed batch inference
ai_action
“we added support for Ray, an open-source compute framework that makes it easy to scale batch inference workloads. By building out a Ray-based distributed task pipeline for batch inference, we were able to optimize resource utilization, e…”
7
CPU inference moves to own data centers
integration
“we moved all of our CPU inference to our own data centers, which gave us more direct control over latency and privacy settings”
8
Custom feature store on Feast and Flink
integration
“we developed our own custom feature store, built on top of the open-source project Feast. Our feature store provided a custom domain-specific language for defining transformations for both batch and streaming features. Flink was adopted …”
9
ML gateway centralizes model access
integration
“We built our unified ML gateway to centralize access to all large models, both open source and internally developed, across a variety of environments, including CPUs and GPUs in the cloud and on premises. Our goal was to create an effici…”
10
vLLM adopted as LLM inference engine
ai_action
“we have adopted vLLM as our primary inference engine for LLMs, leveraging vLLM's high-performance capabilities to power AI applications across Roblox. Since moving to vLLM, we've seen an almost 2x improvement in both latency and throughp…”
Reported outcome

Roblox scaled from fewer than 50 ML inference pipelines to approximately 250, with vLLM delivering an almost 2x improvement in latency and throughput and currently serving approximately 4 billion tokens per week; Avatar Auto Setup produced approximately 8% of UGC avatar bodies as of August 2024.

Reported metrics
ML inference pipelines (early 2023)fewer than 50
ML inference pipelines (current)approximately 250
UGC avatar bodies produced using Avatar Auto Setupapproximately 8%
tokens processed per week (Roblox Assistant)1.5 billion per week
Show all 14 reported metrics
ML inference pipelines (early 2023)fewer than 50
ML inference pipelines (current)approximately 250
UGC avatar bodies produced using Avatar Auto Setupapproximately 8%
tokens processed per week (Roblox Assistant)1.5 billion per week
latency and throughput improvement with vLLMalmost 2x improvement
tokens served per week (vLLM)approximately 4 billion tokens per week
daily personalization requestsapproximately 1 billion
daily active users79.5 million
feature store records ingested per dayapproximately 30 billion
feature store records served per dayapproximately 70 billion
feature store P99 latency50ms
features shared across feature servicesmore than 900
CLIP embedding compute costsignificantly reduced
parallel A/B testsapproximately 20
Reported stack
KubeflowJupyterKServeTriton Inference ServerRayFeastFlinkvLLMvector databaseCLIPRoblox Assistant
Source
https://corp.roblox.com/newsroom/2024/09/running-ai-inference-at-scale-in-the-hybrid-cloud
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Roblox scaled from fewer than 50 ML inference pipelines to approximately 250, with vLLM delivering an almost 2x improvement in latency and throughput and currently serving approximately 4 billion tokens per week; Avat…

What tools did this team use?

Kubeflow, Jupyter, KServe, Triton Inference Server, Ray, Feast, Flink, vLLM, vector database, CLIP.

What results were reported?

ML inference pipelines (early 2023): fewer than 50; ML inference pipelines (current): approximately 250; UGC avatar bodies produced using Avatar Auto Setup: approximately 8%; tokens processed per week (Roblox Assistant): 1.5 billion per week (source-reported, not independently verified).

What failed first in this deployment?

The initial offline inference setup was designed only for real-time sequential workloads, lacked support for task parallelism and multistage processing, and required engineers to write their own data chunking and erro…

How is this back office ops AI workflow structured?

Fragmented teams trigger platform need → Kubeflow and roblox-ml deployed → Feature store enables feature sharing → Extensive pre-release testing → Human moderators review disagreements → Ray enables distributed batch inference → CPU inference moves to own data centers → Custom feature store on Feast and Flink → ML gateway centralizes model access → vLLM adopted as LLM inference engine.

Related back office ops cases