Representative Sampling When Tracking User Satisfaction

chip off the old block

When it comes to surveys, having good quality data encompasses many things like having a high response rate (see blog post here). But it also encompasses having representative sampling. The data should be like a chip off the old block; it should be just like your users as a whole in composition but on a smaller scale.

Surveys have had a long tradition in market research. Often the goal is to profile the larger group in some way– e.g. to report that X% of customers say they would recommend this product to their friends and family. Much effort is placed on ensuring that the data is proportionate to the larger group on respondent dimensions that is thought to matter like age, gender, household size, etc.

Representative sampling is less of a concern in user experience research because survey questionnaires are often used for other purposes like being able to act quickly on customer complaints. But if the purpose is to profile your users, which is often the case when tracking user satisfaction and Net Promoter, you need to make sure your sample represents your users on key dimensions – i.e., on dimensions in which user satisfaction differs. To learn what these dimensions are for your particular website, I suggest you first do a pilot study in which you administer a satisfaction questionnaire, then run segmentation to see on which dimensions does user satisfaction differ.

For example, user satisfaction may vary on these dimensions:
– New users vs. returning users (or frequent vs. less frequent users)
– Geographic location
– Devices and browsers
– Visit duration
– Pageviews

Ideally in the pilot study, you want an equal sample size in each group (as opposed to a representative sample), since the goal is to compare user satisfaction along these dimensions. Let’s say 80% of your traffic comes from returning users; you still want an equal number of returning users and new users in your pilot study in order to compare satisfaction levels between the two groups. Once you find out which dimensions matter for user satisfaction, you can now aim for representative sample on those key dimensions when tracking user satisfaction.

Let’s say you have an online support site, and you run a pilot study on 1,000 users who on average spend little time on your site per visit (so they have a low average visit duration) and 1,000 users who spend much time on your site per visit. Let’s say the former group is more satisfied with your site (likely because they were able to quickly find the information they need). In your actual study, you want your sample to have the same proportion of users with a low average visit duration as your users as a whole. So if 30% of your users have a low average visit duration, 30% of your responses should come from this group.

Having a representative sample and a high response rate are necessary ingredients for high-quality, trustworthy data.