Back office ops · Production

Dropbox accelerates A/B experiment analysis from months to days using machine-learned Expected Revenue (XR) metric

The problem

Dropbox's subscription business made A/B experiment analysis slow and difficult: immediate behavior metrics like file uploads were poorly correlated with user satisfaction, while waiting for actual subscription conversion data took months, limiting how many experiments could be run per year.

First attempt

Four alternative metrics were considered and rejected: immediate activity rate metrics were easy to manipulate and gave false impressions; short-window conversion rates did not account for retention and plan switching; longer-window retention and annual contract value metrics required prohibitively long waits with large data requirements.

Workflow diagram · grounded in source
1
A/B experiment initiated
trigger
“we wanted to measure the effect of a change in how we onboard a new trial user on the first day of their trial”
2
ML models trained for XR
ai_action
“We segment users by trial type and geography, and we use Gradient Boosted Trees algorithms in TensorFlow to train conversion probability and revenue regression models”
3
Daily XR calculation per trial
ai_action
“We calculate XR each day for the first 45 days after a trial starts, and the precision and accuracy of the model improves the more data we collect”
4
XR output calibration
validation
“we perform a calibration as a function of the trial day, the plan type, and the geographical region”
5
Pipeline orchestration
integration
“The orchestration of the daily job is done using Airflow, and the backend is a Hadoop cluster on top of Amazon Web Services (AWS)”
6
Experiment conclusion from XR lift
output
“With machine learning we can now draw accurate conclusions from A/B experiments in a matter of days instead of months”
7
Business health monitoring
feedback_loop
“we can analyze the model to gain insight into drivers of customer intent. We can also monitor the health of our business by tracking XR values across our business weekly”
Reported outcome

Using ML to predict Expected Revenue (XR), Dropbox can draw conclusions from A/B experiments in a matter of days instead of months, enabling more experiments per year, with XR typically within 5% of actual two-year revenue and systematic uncertainty of approximately 3%.

Reported metrics
Experiment conclusion timea matter of days instead of months
XR accuracy vs actual two-year revenuewithin 5%
systematic uncertainty from XR model~3%
absolute difference between XR and actual revenue~10%
Reported stack
TensorFlowAirflowHadoopHiveAmazon S3Spark
Source
https://dropbox.tech/machine-learning/accelerating-our-a-b-experiments-with-machine-learning-xr
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Using ML to predict Expected Revenue (XR), Dropbox can draw conclusions from A/B experiments in a matter of days instead of months, enabling more experiments per year, with XR typically within 5% of actual two-year re…

What tools did this team use?

TensorFlow, Airflow, Hadoop, Hive, Amazon S3, Spark.

What results were reported?

Experiment conclusion time: a matter of days instead of months; XR accuracy vs actual two-year revenue: within 5%; systematic uncertainty from XR model: ~3%; absolute difference between XR and actual revenue: ~10% (source-reported, not independently verified).

What failed first in this deployment?

Four alternative metrics were considered and rejected: immediate activity rate metrics were easy to manipulate and gave false impressions; short-window conversion rates did not account for retention and plan switching…

How is this back office ops AI workflow structured?

A/B experiment initiated → ML models trained for XR → Daily XR calculation per trial → XR output calibration → Pipeline orchestration → Experiment conclusion from XR lift → Business health monitoring.

Related back office ops cases