Back office ops · Production

Virgin Media O2 scales ML production with lean, isolated Vertex AI pipeline container environments

The problem

VMO2's MLOps platform used a single shared container environment for all pipeline tasks, which grew increasingly brittle as the number of data scientists and ML projects expanded — dependency conflicts blocked upgrades, the image ballooned in size slowing node start-up, and local installs exceeded 2 GB.

First attempt

The single monolithic container environment meant all pipeline tasks shared one image — upgrading any package risked breaking other users' projects, dependency conflicts worsened as the platform grew, and the ever-growing image increased download costs and slowed pipeline node start-up.

Workflow diagram · grounded in source

Data scientist triggers ML pipeline

trigger

“The MLOps Platform at VMO2 allows our data scientists and analysts to explore the data, iterate on ML-based solutions and productionise these solutions”

Component routed to lean container

routing

“the container image should be targeted to specific components and only contain the necessary dependencies to execute that piece of code”

Helper code injected at runtime

integration

“make it write out the code for the modules we've specified as dependencies into the corresponding path on the container, and place those files onto the PYTHONPATH”

ML model training runs in isolation

ai_action

“we support a range of libraries such as XGBoost, CatBoost, scikit-learn and TensorFlow”

Models deployed to production

output

“productionise these solutions to make a real impact to enhance the digital experience for our customers”

Reported outcome

Switching to lean, isolated container environments per component type reduced the MLOps package local install from over 2 GB to 147 MB, cut average pipeline running time by approximately 11%, and enabled the production ML model count to grow from 4 to 25 in seven months.

Reported metrics

Pipeline running time~11%

MLOps package local install sizefrom upwards of 2GB down to 147MB

ML models deployed to productionfrom four models to 25 in a matter of seven months

Reported stack

Vertex AI PipelinesKubeflow Pipelines (KFP)BigQueryGCSXGBoostCatBoostscikit-learnTensorFlowDataflowpoetry

Source

https://mlops.community/blog/the-mlops-cookbook-how-we-optimised-our-vertex-ai-pipelines-environments-at-vmo2-for-scale

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Switching to lean, isolated container environments per component type reduced the MLOps package local install from over 2 GB to 147 MB, cut average pipeline running time by approximately 11%, and enabled the productio…

What tools did this team use?

Vertex AI Pipelines, Kubeflow Pipelines (KFP), BigQuery, GCS, XGBoost, CatBoost, scikit-learn, TensorFlow, Dataflow, poetry.

What results were reported?

Pipeline running time: ~11%; MLOps package local install size: from upwards of 2GB down to 147MB; ML models deployed to production: from four models to 25 in a matter of seven months (source-reported, not independently verified).

What failed first in this deployment?

How is this back office ops AI workflow structured?

Data scientist triggers ML pipeline → Component routed to lean container → Helper code injected at runtime → ML model training runs in isolation → Models deployed to production.