Why You Should Combine KPIs in Experiments

Presentation tips - blog5

Imagine a company that sells a line of products and services. This company will likely have multiple goals for its website:

  1. to sell its products and services online
  2. to collect user information for sales prospects
  3. to drive brand awareness and loyalty
  4. to provide online support for existing customers

Let’s say the company has identified 20 KPIs (Key Performance Indicators) that measure the success of these four goals, and it is committed to optimizing the conversion of these goals by running many experiments. Should the company launch the treatment if some KPIs perform better but others perform worse than the control (i.e. the original site)?

Continue reading

5 Steps to Compare Multiple KPIs

Presentation tips - blog4

To know how your website’s doing, you need to define your Key Performance Indicators. If you’re running a blog (or some sort of content publishing site), and your goal is to increase user engagement, your Key Performance Indicators may include number of visits, pageviews per visit, and visit duration. 

Continue reading

Research Lessons from American Idol

keith-american-idol_400

I haven’t watched American Idol in years, but I still remember the cringe-worthy auditions of those who claim to be the next Idol yet can’t carry a tune to save their lives. For some, it’s hard to objectively evaluate their own talent when there is so much at stake.

Psychologists have coined the term “motivated reasoning,” a tendency for people to reason in ways that allow them to form or maintain desirable beliefs (e.g. that they can sing). They may readily accept information that supports their beliefs as valid but question information that challenges their beliefs (remember how angry those contestants were at the Idol judges?).

In a similar vein, research should not be conducted by those who have a stake in how the research findings turn out. This may seem obvious, but I’m surprised by how often it still happens.

Continue reading