Back office ops · Production

Canva rebuilds GPU-accelerated ML container infrastructure with Kubernetes and Nix

The problem

Canva's ML Platform team needed to rebuild their cloud GPU container base images from scratch using Nix, but their initial images failed to find the GPU at runtime, and the root cause was non-obvious across a complex stack of OS, drivers, container runtime, and image layers.

First attempt

The initial Nix-built base images failed for three distinct reasons: the required NVIDIA_DRIVER_CAPABILITIES environment variable was absent, preventing the NVIDIA container runtime from mounting driver libraries; the distroless Nix container's LD_LIBRARY_PATH was incompatible with the Amazon Linux 2 host; and the images shipped without the dynamic linker, causing mapped NVIDIA utilities to report 'No such file or directory'.

Workflow diagram · grounded in source
1
ML workload submitted to cluster
trigger
“When an ML platform user submits this program to our cluster”
2
GPU device allocation
integration
“it looks for the matching device plugin — in this case k8s-device-plugin — and initiates an 'allocation phase' within which the device plugin sets up the container with the GPU devices, mapping them into /dev/ in the container's filesystem”
3
NVIDIA driver library mount
integration
“a base image must set NVIDIA_DRIVER_CAPABILITIES=compute,utility in its environment because the NVIDIA container runtime library looks at this to determine which libraries to mount inside a new container”
4
GPU-accelerated ML execution
ai_action
“We use ML to remove image backgrounds and sharpen our core recommendation, searching, and personalization capabilities”
Reported outcome

Canva successfully built Nix-based GPU container images that correctly mount NVIDIA driver files and support GPU-accelerated ML workloads in production on Kubernetes, with the stack running in production for over 12 months.

Reported metrics
Production runtimeover 12 months
PyTorch base image sizeabout 80MB
Reported stack
KubernetesPyTorchTensorFlowCUDA
Source
https://www.canva.dev/blog/engineering/supporting-gpu-accelerated-machine-learning-with-kubernetes-and-nix/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Canva successfully built Nix-based GPU container images that correctly mount NVIDIA driver files and support GPU-accelerated ML workloads in production on Kubernetes, with the stack running in production for over 12 m…

What tools did this team use?

Kubernetes, PyTorch, TensorFlow, CUDA.

What results were reported?

Production runtime: over 12 months; PyTorch base image size: about 80MB (source-reported, not independently verified).

What failed first in this deployment?

The initial Nix-built base images failed for three distinct reasons: the required NVIDIA_DRIVER_CAPABILITIES environment variable was absent, preventing the NVIDIA container runtime from mounting driver libraries; the…

How is this back office ops AI workflow structured?

ML workload submitted to cluster → GPU device allocation → NVIDIA driver library mount → GPU-accelerated ML execution.

Related back office ops cases