Research Lessons from American Idol


I haven’t watched American Idol in years, but I still remember the cringe-worthy auditions of those who claim to be the next Idol yet can’t carry a tune to save their lives. For some, it’s hard to objectively evaluate their own talent when there is so much at stake.

Psychologists have coined the term “motivated reasoning,” a tendency for people to reason in ways that allow them to form or maintain desirable beliefs (e.g. that they can sing). They may readily accept information that supports their beliefs as valid but question information that challenges their beliefs (remember how angry those contestants were at the Idol judges?).

In a similar vein, research should not be conducted by those who have a stake in how the research findings turn out. This may seem obvious, but I’m surprised by how often it still happens.

If you’re a VP of a website company, and a product manager has found research evidence that there’s user demand for a certain product idea she’s been proposing, and she’d get nice funding if her idea gets approved, you should carefully evaluate this evidence. Did she conduct a survey? A focus group? One-on-one interviews? Who were the participants? What questions did she ask? Did she even choose the right research method given the research objective?

I’m not claiming that people often falsify research findings; doing so is the exception rather than the norm. But motivated reasoning is often an unintentional process. The product manager may be honestly trying to test her product idea in an objective manner, but she may be less diligent in quality control when her data analysis supports rather than refutes her product idea without her realizing it.

Outsourcing research to third party suppliers (i.e. consultants) can often help produce more objective and accurate research, but not always. Third party suppliers may also have a stake in how the research findings turn out if it’s the product manager they report to, and they’d get more research contracts from the product manager if they produce research findings that support the product manager’s ideas. When working with research suppliers, it’s better if they can report to someone who is not motivated to reach certain conclusions so that they are free to work as objective, third party members.

Thankfully, many research suppliers would not take on unethical research projects like these, as they have their own reputation to maintain, but it just takes one unethical research supplier to give all research suppliers a bad name.