Some people believe that “bigger is better” when it comes to sample size – the more survey respondents you have, the more trustworthy your results.
True, a bigger sample gives you more precise estimates, which is necessary for your results to be trustworthy. It also gives you more statistical power to detect differences between estimates and a benchmark, or differences between control vs. treatment.
But a bigger sample is only necessary and not sufficient for results to be trustworthy. You also need to correct for nonresponse error, or the bias in survey results due to non-respondents having different characteristics from survey respondents.
#1 – Only survey those you want to survey.
- I recently got a survey asking me to rate my satisfaction with one of the website’s products even though I haven’t used it; I’ve only read about the product on their website.
- In a user satisfaction survey, only survey those who’ve used your product, or at least make a response option like “Not applicable” available. Better yet, survey those who’ve used your product several times so that they have more of an informed opinion.
- Sounds obvious, but the website that made this mistake is arguably the most well-known website in the world. Sometimes it’s the obvious that gets overlooked.