Back office ops · Production

Canva recommendation system: handling empty results, irrelevant outputs, and production failures at 60M+ user scale

The problem

Canva's personalization system serving over 60 million monthly active users faces two recurring failure classes: unexpected results (empty recommendations from cold-start or low model confidence, and irrelevant outputs from model imperfections) and failure to respond (high latency from large deep learning models and horizontal scaling limits hit during peak traffic while most engineers are asleep in Australia).

First attempt

Recommendation models have produced no results or irrelevant results; horizontal scaling limits have been hit multiple times due to Canva's fast-growing user base or new models requiring larger machines; and some models take around 15 to 20 minutes to scale, making roll-forward during incidents impractical.

Workflow diagram · grounded in source
1
User template interaction triggers cycle
trigger
“Every time a user interacts with a template, in the background, we call the recommendation models and update the caches for that user with the new recommendations”
2
Recommendation model inference
ai_action
“we call the recommendation models and update the caches for that user with the new recommendations”
3
Validation and filtering
validation
“These steps include filtering out duplicates and filtering out malformed results. Malformed results include invalid result IDs or broken metadata when these IDs are converted into actual recommendation objects (the image URL might be mal…”
4
Fallback routing on empty or low-confidence results
routing
“we fall back to broad recommendations, which are locale and platform-specific”
5
Near-line cache update
feedback_loop
“we introduced near-line inference into our service. This means actively updating recommendations in the background and storing them in a caching layer while the user interacts with the system”
6
On-call incident response
human_review
“the two main ways to handle the situation are either a rollback or a switch-off. We generally prefer to roll back whenever possible because we still provide the benefit of the model to our users”
Reported outcome

Canva mitigates recommendation failures through locale- and platform-specific fallbacks, near-line inference caching to keep recommendations reactive to user interactions, metric-threshold deployment gates, visual model reports for debugging, auto-scaling policies, and independent per-model controllers enabling rollback or switch-off during incidents without affecting other models.

Reported metrics
Monthly active usersmore than 60 million
Model scale-up timearound 15 to 20 minutes
Reported stack
deep learning modelscaching layer
Source
https://www.canva.dev/blog/engineering/recommender-systems-when-they-fail-who-are-you-gonna-call/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Canva mitigates recommendation failures through locale- and platform-specific fallbacks, near-line inference caching to keep recommendations reactive to user interactions, metric-threshold deployment gates, visual mod…

What tools did this team use?

deep learning models, caching layer.

What results were reported?

Monthly active users: more than 60 million; Model scale-up time: around 15 to 20 minutes (source-reported, not independently verified).

What failed first in this deployment?

Recommendation models have produced no results or irrelevant results; horizontal scaling limits have been hit multiple times due to Canva's fast-growing user base or new models requiring larger machines; and some mode…

How is this back office ops AI workflow structured?

User template interaction triggers cycle → Recommendation model inference → Validation and filtering → Fallback routing on empty or low-confidence results → Near-line cache update → On-call incident response.

Related back office ops cases