Quality assurance · Production

Canva builds synthetic data evaluation pipeline to improve private design search without accessing user data

The problem

Canva's engineers could only test private design search changes through a handful of manual queries on their own accounts due to strict privacy constraints preventing access to real user designs or queries, then had to wait days for online A/B experiments to validate changes.

First attempt

Limited offline testing had low statistical power to catch poorly performing changes, and progressing quickly to online experiments risked exposing real users to degraded search behavior.

Workflow diagram · grounded in source
1
Generate synthetic design content
ai_action
“we seeded GPT-4o with a realistic topic for a design. We also specified a design type, such as document, presentation, or Instagram post, which was sampled from the distribution of real Canva design types, and prompted GPT-4o to brainsto…”
2
Generate evaluation queries
ai_action
“Queries were then generated from these relevant designs using a combination of programmatic techniques and rewriting by GPT-4o. For example, an easy query might be created by sampling one or more words from the design's title or content.…”
3
Generate nonrelevant designs
ai_action
“GPT-4o generated designs containing some, but not all, of the words from the query. This approach ensures that the nonrelevant designs have a lower level of text match to the query compared to the relevant design”
4
Run local search pipeline
integration
“building a pipeline in which we combine externally supported containers for components such as ElasticSearch, with our own internal Testcontainers, run with an exact replica of our production configurations”
5
Calculate recall and precision
output
“We then run the local search pipeline on this state with the test query and pass the results to our evaluation module to calculate recall and precision metrics”
6
Visualize results in Streamlit
output
“we built our own custom visualization tool using Streamlit. This tool allows for comparison of the aggregated recall and precision metrics for different configurations side by side, as well as segmenting the performance by different quer…”
Reported outcome

The synthetic evaluation pipeline produces fully reproducible results on more than 1000 test cases in under 10 minutes, enabling more than 300 offline evaluations in the same time a single online experiment takes, all without accessing any real user data.

Reported metrics
Offline evaluation speedmore than 1000 test cases in less than 10 minutes
Evaluation reproducibilitycompletely reproducible evaluation results within minutes
Reported stack
GPT-4oLLMsTestcontainersElasticSearchStreamlit
Source
https://www.canva.dev/blog/engineering/how-to-improve-search-without-looking-at-queries-or-results/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The synthetic evaluation pipeline produces fully reproducible results on more than 1000 test cases in under 10 minutes, enabling more than 300 offline evaluations in the same time a single online experiment takes, all…

What tools did this team use?

GPT-4o, LLMs, Testcontainers, ElasticSearch, Streamlit.

What results were reported?

Offline evaluation speed: more than 1000 test cases in less than 10 minutes; Evaluation reproducibility: completely reproducible evaluation results within minutes (source-reported, not independently verified).

What failed first in this deployment?

Limited offline testing had low statistical power to catch poorly performing changes, and progressing quickly to online experiments risked exposing real users to degraded search behavior.

How is this quality assurance AI workflow structured?

Generate synthetic design content → Generate evaluation queries → Generate nonrelevant designs → Run local search pipeline → Calculate recall and precision → Visualize results in Streamlit.

Related quality assurance cases