Conducting controlled experiments is the best way of determining whether a site or app redesign would lead to improvements on key metrics. One barrier is the amount of time or resources it takes to run experiments. You may have a low traffic site, you may want to detect small differences in key metrics (i.e. fractions of a percent), or you may want to get experiment results faster. Here are some suggestions on how to run experiments more efficiently.
Tag Archives: power
What if my sample size is not big enough for my experiment?
My main recommendation – include only users impacted by the change in your analysis; exclude users who are not.
- Let’s say you have an e-commerce site. You want to test whether certain changes to your checkout page would increase conversion (% of users purchasing).
- You want to run a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups.
- Your current conversion is 5%; you want to detect conversion changes as small as 10% (with the conventional 80% probability of detection and confidence level at 95%).
- According to this table in my blog post, you would need 30,400 users in each group, or 30400 x 4 = 121,600 users in total visiting your site. (That’s a lot!)
3 Steps to Calculate Sample Sizes for Experiments
To calculate how many people you need in your experiment, you need to know 3 things:
1. How many groups are in your experiment?
- In an A/B experiment with a control and treatment group, you have 2 groups.
- In a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups, you have 4 groups.
- The more groups you have, the more people you need.
Continue reading
