# 3 Steps to Calculate Sample Sizes for Experiments

To calculate how many people you need in your experiment, you need to know 3 things:

1. How many groups are in your experiment?

• In an A/B experiment with a control and treatment group, you have 2 groups.
• In a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups, you have 4 groups.
• The more groups you have, the more people you need.

2. What is your current conversion?

• If right now 5% of users visiting your e-commerce site end up purchasing, and your experiment is to see if a treatment would increase purchasing, your current conversion is 5%.
• If 20% of your users click on an ad on your website, and you want to see if a treatment would increase ad clicks, your conversion is 20%.
• The lower your current conversion, the more people you need in your experiment.

3. How big would the difference in conversion be for you to care?

• We all want to detect differences in conversion between the experimental groups no matter how small, but tiny differences require large samples to detect.
• The optimal sample size allows you to detect the smallest difference worth detecting.

Table: Number of users you need in your experiment per group.*

• For e.g., your current conversion is 5%, and you want to detect conversion changes as small as 10%.
• In an A/B experiment, you’ll need 60,800 users in your experiment, or 30,400 users in each of the 2 groups.
• In a 2 x 2 Multi-Variable experiment, you’ll need 121,600 users in your experiment, or 30,400 users in each of the 4 groups.

* This sample size will give you a reasonable (80%) chance of detecting the smallest conversion change you want to be able to detect while holding confidence level at the conventional 95%.

Coming soon: What if my sample size is not big enough for my experiment?

Source: Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne, Controlled Experiments on the Web: Survey and Practical Guide, Data Mining and Knowledge Discovery journal, Vol 18(1), p. 140-181, 2009. DOI.