Imagine a company that sells a line of products and services. This company will likely have multiple goals for its website:
- to sell its products and services online
- to collect user information for sales prospects
- to drive brand awareness and loyalty
- to provide online support for existing customers
Let’s say the company has identified 20 KPIs (Key Performance Indicators) that measure the success of these four goals, and it is committed to optimizing the conversion of these goals by running many experiments. Should the company launch the treatment if some KPIs perform better but others perform worse than the control (i.e. the original site)?
To know how your website’s doing, you need to define your Key Performance Indicators. If you’re running a blog (or some sort of content publishing site), and your goal is to increase user engagement, your Key Performance Indicators may include number of visits, pageviews per visit, and visit duration.
I haven’t watched American Idol in years, but I still remember the cringe-worthy auditions of those who claim to be the next Idol yet can’t carry a tune to save their lives. For some, it’s hard to objectively evaluate their own talent when there is so much at stake.
Psychologists have coined the term “motivated reasoning,” a tendency for people to reason in ways that allow them to form or maintain desirable beliefs (e.g. that they can sing). They may readily accept information that supports their beliefs as valid but question information that challenges their beliefs (remember how angry those contestants were at the Idol judges?).
In a similar vein, research should not be conducted by those who have a stake in how the research findings turn out. This may seem obvious, but I’m surprised by how often it still happens.