To calculate how many people you need in your experiment, you need to know 3 things:
1. How many groups are in your experiment?
- In an A/B experiment with a control and treatment group, you have 2 groups.
- In a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups, you have 4 groups.
- The more groups you have, the more people you need.
2. What is your current conversion?
- If right now 5% of users visiting your e-commerce site end up purchasing, and your experiment is to see if a treatment would increase purchasing, your current conversion is 5%.
- If 20% of your users click on an ad on your website, and you want to see if a treatment would increase ad clicks, your conversion is 20%.
- The lower your current conversion, the more people you need in your experiment.
3. How big would the difference in conversion be for you to care?
- We all want to detect differences in conversion between the experimental groups no matter how small, but tiny differences require large samples to detect.
- The optimal sample size allows you to detect the smallest difference worth detecting.
Table: Number of users you need in your experiment per group.*
- For e.g., your current conversion is 5%, and you want to detect conversion changes as small as 10%.
- In an A/B experiment, you’ll need 60,800 users in your experiment, or 30,400 users in each of the 2 groups.
- In a 2 x 2 Multi-Variable experiment, you’ll need 121,600 users in your experiment, or 30,400 users in each of the 4 groups.
* This sample size will give you a reasonable (80%) chance of detecting the smallest conversion change you want to be able to detect while holding confidence level at the conventional 95%.
Coming soon: What if my sample size is not big enough for my experiment?
Source: Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne, Controlled Experiments on the Web: Survey and Practical Guide, Data Mining and Knowledge Discovery journal, Vol 18(1), p. 140-181, 2009. DOI.