Four Suggestions for Running Experiments More Efficiently

Conducting controlled experiments is the best way of determining whether a site or app redesign would lead to improvements on key metrics. One barrier is the amount of time or resources it takes to run experiments. You may have a low traffic site, you may want to detect small differences in key metrics (i.e. fractions of a percent), or you may want to get experiment results faster. Here are some suggestions on how to run experiments more efficiently.

Continue reading

Advertisements

What if my sample size is not big enough for my experiment?

My main recommendation – include only users impacted by the change in your analysis; exclude users who are not.

  • Let’s say you have an e-commerce site. You want to test whether certain changes to your checkout page would increase conversion (% of users purchasing).
  • You want to run a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups.
  • Your current conversion is 5%; you want to detect conversion changes as small as 10% (with the conventional 80% probability of detection and confidence level at 95%).
  • According to this table in my blog post, you would need 30,400 users in each group, or 30400 x 4 = 121,600 users in total visiting your site. (That’s a lot!)

Continue reading

3 Steps to Calculate Sample Sizes for Experiments

To calculate how many people you need in your experiment, you need to know 3 things:

1. How many groups are in your experiment?

  • In an A/B experiment with a control and treatment group, you have 2 groups.
  • In a 2 x 2 Multi-Variable experiment with 1 control and 3 treatment groups, you have 4 groups.
  • The more groups you have, the more people you need.
    Continue reading