3 Steps to Calculate Sample Sizes for Experiments

To calculate how many people you need in your experiment, you need to know 3 things:

1. How many groups are in your experiment?

  • In an A/B experiment with a control and treatment group, you have 2 groups.
  • In a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups, you have 4 groups.
  • The more groups you have, the more people you need.

2. What is your current conversion?

  • If right now 5% of users visiting your e-commerce site end up purchasing, and your experiment is to see if a treatment would increase purchasing, your current conversion is 5%.
  • If 20% of your users click on an ad on your website, and you want to see if a treatment would increase ad clicks, your conversion is 20%.
  • The lower your current conversion, the more people you need in your experiment.

3. How big would the difference in conversion be for you to care?

  • We all want to detect differences in conversion between the experimental groups no matter how small, but tiny differences require large samples to detect.
  • The optimal sample size allows you to detect the smallest difference worth detecting.

Table: Number of users you need in your experiment per group.*

Sample size table

  • For e.g., your current conversion is 5%, and you want to detect conversion changes as small as 10%.
  • In an A/B experiment, you’ll need 60,800 users in your experiment, or 30,400 users in each of the 2 groups.
  • In a 2 x 2 Multi-Variable experiment, you’ll need 121,600 users in your experiment, or 30,400 users in each of the 4 groups.

* This sample size will give you a reasonable (80%) chance of detecting the smallest conversion change you want to be able to detect while holding confidence level at the conventional 95%.

Coming soon: What if my sample size is not big enough for my experiment?

Source: Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne, Controlled Experiments on the Web: Survey and Practical Guide, Data Mining and Knowledge Discovery journal, Vol 18(1), p. 140-181, 2009. DOI.

Advertisements

One thought on “3 Steps to Calculate Sample Sizes for Experiments

  1. Pingback: What if my sample size is not big enough for my experiment? | User Experience Research

Comments are closed.