My main recommendation – include only users impacted by the change in your analysis; exclude users who are not.
- Let’s say you have an e-commerce site. You want to test whether certain changes to your checkout page would increase conversion (% of users purchasing).
- You want to run a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups.
- Your current conversion is 5%; you want to detect conversion changes as small as 10% (with the conventional 80% probability of detection and confidence level at 95%).
- According to this table in my blog post, you would need 30,400 users in each group, or 30400 x 4 = 121,600 users in total visiting your site. (That’s a lot!)
But let’s say not all users are exposed to your checkout page. Only those who initiate checkout see it. You can reduce the sample size needed for your experiment if you include only users who see the checkout page in your analysis.
- Let’s say only 20% of users initiate checkout and see your checkout page. Among these users, 25% complete the purchase.
- Your current conversion for this group is now 25%, much higher than the 5% current conversion of all users visiting your site, and higher current conversion means a smaller sample is needed.
- At 25% current conversion, you only need 4,800 users in each group, or 4,800 x 4 = 19,200 users who initiate checkout to detect conversion changes as small as 10% (see my blog post for more info).
- OK, so you only need 19,200 users in your analysis. So how many users do you need visiting your site in total to run your experiment?
- Given that 20% of users visiting your site initiate checkout, the total number of users you need visiting your site to run your experiment is 19,200 / 0.20 = 96,000.
- By filtering out from your analysis the 80% of users not impacted by the change in checkout, you’ve reduced the sample size required for the experiment by 121,600 – 96,000 = 25,600 users.
NOTE: You should run your experiment for at least a week, even if you can get a big enough sample in less than a week for your experiment. Conversions like purchasing can often depend on the day and time in which users are visiting your site.
More recommendations to come in later posts.