magazine issue 3 2015 / Issue 25

The perverse incentives that stand as a roadblock to scientific reform

• written by Brent W. Roberts
• edited by Sam Portnow

In one respect it is hard to argue with this motive.  We hold those scientists who bring unique discoveries to their field in the highest esteem.  And, every once in a while, someone actually does do something truly innovative. In the mean time, we get caught up in the pursuit of cutesy counter-intuitiveness all under the hope that our predilection will become the next big innovation.  To be clear, it is really cool when researchers identify something counter-intuitive about human behavior. But the singular pursuit of such goals often leads us to ignore the enduring questions of the human dilemma. 

The Tyranny of the Clean Narrative

The last piece of the incentive structure is quite possibly the most insidious because everyone pushes it—authors, reviewers, and editors alike.  To be successful your research must provide a clean narrative.  The research story must have a consistent beginning, middle, and end.  This means, the intro should correspond perfectly to the method section, which correlates perfectly with the findings, which all have to be statistically significant and confirm the hypotheses stated in the introduction.  The powerful incentive for a clean narrative promotes many of the questionable research practices we use.  We HARK (Hypothesizing After the Results are Known) so as to make a clean narrative.  We throw out null findings or are told to throw out null findings by reviewers and editors in order to achieve a clean narrative.  We avoid admitting to failures to replicate, again, because it would undermine a clean narrative.

The bias towards a clean narrative is especially prominent at our most prestigious journals.  Our top journals envision themselves as the repository for impressive new discoveries.  New discoveries cannot possess blemishes.  Prioritizing a clean narrative leads reviewers and editors to act as gatekeepers and mistakenly recommend against publishing studies that have null effects.  For that matter, when we as researchers fail to put together a consistent package of studies we usually self-select the paper into a lower tier journal because we know it won’t be received well at the top outlets. That means that our most honest science is most likely in our “worst” journals because they have a tendency to be more forgiving of messy narratives. 

Summary

In sum, these four pillars of perverse incentives stand strong against efforts to make our science more transparent and reproducible.  Arguments against these changes, by their nature conservative arguments to keep the status quo, only help to perpetuate a system that has rewarded individuals and individual careers, but has undermined the integrity and reliability of our science.  Reporting only statistically significant findings results in a literature that does not represent the truth. Pushing small N, conceptual replications aids and abets the hiding of inelegant findings that don’t conform perfectly to the theories we test.  Overvaluing counter-intuitive findings undermines the development of cumulative knowledge that might be relied on for social policy.  Policing studies so that they only report “clean findings” and thus have a clean narrative further promotes a depiction of science that is too good to be true.

Reasons for Pessimism

For some, the open science movement and the efforts of specific journals to change the parameters of the publication process give rise to the hope that our scientific products will become more reliable (Vazire, 2014).  I am pessimistic about our ability to change the existing system for one overarching reason.  The proposed changes of the open science and reproducibility movement are largely perceived as punitive.  They do not provide an alternative, compelling reward structure, but instead instigate a largely corrective check on existing practices. 

And, it is difficult to see how it can be any other way.  The existing system has maximized our success.  It provides a weird p-value driven utopia.  With the infinite flexibility of our current incentive system we can, as Simmons et al (2011) showed, provide empirical evidence for any idea, no matter how absurd it may be. All it takes is a lot of data and analyses. I fear these incentives have led to a system in which many, many more people have succeeded for ideas that will not last the test of time or replication, whichever comes first.  In other words, we have an excess of success.

The problem with the current push for methodological reform is that, it is hard, unrewarding, and a will result in a science that is a lot uglier than our current system. The truth is less elegant than what we produce in our scientific journals. Adopting a sounder approach to our scientific methods, which is critical for our long-term viability as a science, will inevitably curtail our excess success. There will be fewer famous psychologists, fewer book contracts, and fewer Ted Talks.

This is one reason why people fight so strongly against the benign reforms being proposed.  We’ve had a good gig for a long time and the future will be less bright if we do things with transparency and reproducibility in mind.  Of course, it is quite possible that our future will be less bright either way.

Relatedly, my pessimism is deepened by watching a critical mass of the senior leadership in psychology protest the proposed changes of the reform movement.  On one hand, I can understand their reticence to change.  The current system has obviously served them well.  After all, they are some of the most eminent, successful researchers in our guild.