Dropbox accelerates A/B experiment analysis from months to days using machine-learned Expected Revenue (XR) metric
Dropbox's subscription business made A/B experiment analysis slow and difficult: immediate behavior metrics like file uploads were poorly correlated with user satisfaction, while waiting for actual subscription conversion data took months, limiting how many experiments could be run per year.
Using ML to predict Expected Revenue (XR), Dropbox can draw conclusions from A/B experiments in a matter of days instead of months, enabling more experiments per year, with XR typically within 5% of actual two-year revenue and systematic uncertainty of approximately 3%.
Four alternative metrics were considered and rejected: immediate activity rate metrics were easy to manipulate and gave false impressions; short-window conversion rates did not account for retention and plan switching; longer-window retention and annual contract value metrics required prohibitively long waits with large data requirements.
https://dropbox.tech/machine-learning/accelerating-our-a-b-experiments-with-machine-learning-xr