Back office ops · Production
Canva's distributed ML hyperparameter optimization with Argo Workflows and Bayesian optimization
The problem
Canva's ML teams faced exponentially growing hyperparameter search spaces and severe resource constraints from vertical scaling, making tuning large models impractically slow.
Workflow diagram · grounded in source
1
Engineer configures search space
trigger
“Argo CLI and UI enable machine learning engineers to specify their desired search spaces and hyperparameter configurations at run-time”
2
Bayesian optimizer generates batch
ai_action
“By building a probabilistic "surrogate" model from hyperparameter values and previous model evaluation results, Bayesian optimization methods balance exploring regions in the search space with high uncertainty while exploiting regions ar…”
3
Fan-out to parallel model trainers
integration
“The workflow then fans out the batch of hyperparameters to each model trainer running in parallel”
4
Parallel model training
ai_action
“Model Trainers: A batch of model trainer containers that accepts hyperparameter values and returns pre-defined evaluation metrics”
5
Aggregate results and check termination
validation
“Argo then aggregates the values into the next instantiation, beginning the next iteration if the termination criteria is not satisfied”
6
Warm-start from prior job state
feedback_loop
“engineers can also create new hyperparameter optimization jobs from the state of a previous one. This effectively enables engineers to warm-start the optimization and iteratively relax or constrain the search space across multiple jobs”
Reported outcome
The distributed HPO system delivered an average speedup of at least five times over the previous process, cutting optimization time from over a week to a little over a day.
Reported metrics
Optimization speedupat least five times
Optimization timefrom over a week to a little over a day
Reported stack
Kubernetes
Source
https://www.canva.dev/blog/engineering/machine-learning-hyperparameter-optimization-with-argo/
Read source ↗Frequently asked questions
What did this team achieve with this AI workflow?
The distributed HPO system delivered an average speedup of at least five times over the previous process, cutting optimization time from over a week to a little over a day.
What tools did this team use?
Kubernetes.
What results were reported?
Optimization speedup: at least five times; Optimization time: from over a week to a little over a day (source-reported, not independently verified).
How is this back office ops AI workflow structured?
Engineer configures search space → Bayesian optimizer generates batch → Fan-out to parallel model trainers → Parallel model training → Aggregate results and check termination → Warm-start from prior job state.