5 Steps to Compare Multiple KPIs

Presentation tips - blog4

To know how your website’s doing, you need to define your Key Performance Indicators. If you’re running a blog (or some sort of content publishing site), and your goal is to increase user engagement, your Key Performance Indicators may include number of visits, pageviews per visit, and visit duration. 

Let’s say you have come up with a design change for your website, and you run an A/B experiment to see if implementing the design change would increase user engagement. After running your experiment for two weeks, you get mixed results. Your treatment (i.e. design change) got users to visit your site more often, but it also led users to view less pages and spend less time with your site per visit.

Average visits per user

Average pageviews

Average visit duration (in seconds)

Control

2.5

2.8

885

Treatment

4.5

1.9

625

So is the treatment better, worse, or the same as your control (i.e. the original site)?

Answering the question is a lot easier if you could combine the 3 KPIs into a single metric to compare the performance of the control vs. treatment. To do this, you’ll need to
(a) convert your KPIs so that they share the same measurement unit, and
(b) combine your converted KPIs into a single metric (see Roy, 2001 p. 50). Here’s how:

Step #1: Determine “Worst Value” and “Best Value” of each KPI

  • KPI#1: Visits per user
    • Worst Value = 1
    • (A Worst Value of 1 means the user visited your site only once during the 2-week experiment. It can’t be 0 because the user needs to visit your site at least once to be in your experiment.)
    • Best Value = 14
    • (The Best Value is something you choose depending on your historical data. Let’s say your most frequent users are visiting your site almost everyday, but not quite. Setting the Best Value to 14 means setting the Best Value to daily visits from users during your 2-week experiment.)
  • KPI#2: Pageviews
    • Worst Value = 1
    • (1 is the Worst Value for Pageviews because it means the user landed onto your site and left without any interaction.)
    • Best Value = 20
    • (The Best Value can also be what makes sense depending on your site. If you have 20 pages on your site, setting the Best value to 20 would mean the user viewed every page of your site.)
  • KPI#3: Visit duration
    • Worst value = 0 seconds
    • (0 is the Worst Value because it means the user landed onto your site and left without any interaction.)
    • Best value = 3,600 seconds
    • (This best value is appropriate if historical data indicates that users rarely spend more than an hour interacting with your site in one visit.)

Step #2: Convert the KPIs

  • Let’s say you had 10,000 users visiting your site during your 2-week experiment. Let’s examine User #1, User #2, and User #3.

User #1

User #2

User #3

Number of visits

2

4

5

Average pageviews

3

2

1

Average visit duration

900 sec

600 sec

0 sec

  • For User #1, convert the 3 KPIs by applying the formula:

Converted KPI formula

Converted KPI User1

  • All Converted KPIs will range from 0 to 1, with higher values meaning greater user engagement
  • Convert the 3 KPIs for User #2, User #3 … all the way to User #10,000

Step #3: Determine the Importance of each KPI

  • Depending on your goals for your site, you may not value each KPI equally. You may care about “number of visits” and “pageviews” more than “visit duration.”
  • You should assign more influence or “weight” to the KPIs you care about
  • E.g. 40% of your single metric can come from “number of visits”, 40% from “pageviews”, and 20% from “visit duration”

Step #4: Weight the Converted KPIs

  • E.g. Weighting the Converted KPIs for User #1

Weighted KPI

  • Do this for User #2, User #3 … all the way to User #10,000

Step #5: Sum the Weighted KPIs into a Single Metric

  • E.g. For User #1, she will have a single score of 3.20 + 4.40 + 5.00 = 12.6.
  • Do the same thing for User #2, User #3 … all the way to User #10,000 so that every user will have a single score
  • The single metric will range from 0 to 100, with higher scores meaning greater user engagement

You now have a single metric (also called Overall Evaluation Criterion) that takes into account the 3 user engagement KPIs.

This single metric will allow you to compare the treatment vs. the control and help you decide whether your website redesign is better, worse, or the same as the original. Read my blog post “Why You Should Combine KPIs in Experiments” to learn more about the benefits of using Overall Evaluation Criterion.

Reference: Roy, Ranjit K. 2001. Design of Experiments using the Taguchi Approach: 16 Steps to Product and Process Improvement. S.1.: John Wiley & Sons, Inc, 2001. 0-471-36101-1.

About these ads

One thought on “5 Steps to Compare Multiple KPIs

  1. Pingback: Why You Should Combine KPIs in Experiments | User Experience Research

Comments are closed.