The science of psychology is in crisis.
One by one, many of its flashiest and most famous results have collapsed in the last decade as a new generation of researchers have re-examined famous findings.
Forcing a smile can’t really make you happy. The smell of chocolate cookies doesn’t make you a better test taker. A robot named “statcheck” is trawling through published papers for statistical errors, and posting public comments when it finds them.
It’s a scary time to be a member of the psychological academic establishment. As the social psychologist Michael Inzlicht wrote on his blog back in February, “Our problems are not small and they will not be remedied by small fixes. Our problems are systemic and they are at the core of how we conduct our science.”
Plenty of research seems to still sit on a firm ground of good methods. But so many of psychology’s most astonishing (and widely-reported) findings have turned over in their tanks in the last several years that it’s hard to imagine more won’t follow.
That’s the context for a fairly shocking letter making the rounds on Twitter in psychology circles right now. In it, Susan Fiske, the former president of the Association for Psychological Science and a respected researcher at Princeton, attacks unnamed “adversaries” of psychology. She calls them names like “methodological terrorist” and writes that their criticism of fellow researchers should happen in private, in respected journals, or not at all.
It’s a major volley, fired without warning in the cold war between those who built up modern psychology and those who she alleges have begun to tear them down.
In it, Fiske accuses people who’ve publicly criticized the methods of psychological research of public bullying and what she calls “methodological terrorism.” Not surprisingly, the line in the letter with this phrase has gotten the most attention.
Fiske is referring to a group of people that she sees as dangerous and fanatical. She accuses them of “terrorism” to suggest that their tactics – publishing criticisms on Twitter and Facebook – are outside of the rules of war that have stood for years in the field of psychological research. She also calls this loosely-defined group the “self-appointed data police” and argues that their social media activism hurts careers and undermines science. She writes that their conclusions, often published without peer-review, are suspect and that they’ve so poisoned the climate of psychology as to drive people out of the field. She doesn’t name names.
Who gets to criticize science?
If you followed the crisis that struck journalism about a decade ago, with the rise of online journalism and blogging and collapse of print media, that’s a good analogy for what’s happening here.
An old guard is lashing out at a strange new cohort that fails to respect their culture and norms. The old guard has decades of institutional power and prestige to uphold, and controls the most-respected means of distributing information. The new cohort is snarky, irreverent, and dodges their betters’ control with social media and blogs.
In the days since the leak, people have publicly challenged Fiske at a conference and criticized her online:
For shame @PsychScience: Susan Fiske’s column is inappropriate in both tone and content for APS Observer. Where’s the editorial oversight?
— Louise Connell (@DrLouiseConnell) September 20, 2016
Re: Fiske’s @APS column: I’ve never read anything about a failed replication attempt that’s as vicious as what she wrote.
— Candice Coker Morey (@CandiceMorey) September 20, 2016
Susan Fiske invoking the concept of terrorism at all in this context is inflammatory & completely outrageous.
— Chris Chambers (@chrisdc77) September 19, 2016
The essential question here is who psychology belongs to. How do scientists weigh the dangers of damaging their peers’ careers against the drive to weed out bad science?
‘She prefers moderated forums where criticism is done in private. I prefer open discussion.’
Andrew Gelman, a Columbia University statistician and political scientist, wrote a blog post responding to Fiske. He makes again the case for the replication crisis, then alleges that Fiske’s personal reputation is so bound up in old, faulty research that it incentivizes her to deny the truth. And he argues that public debates and criticism are good and healthy for psychology.
“Who is Susan Fiske and why does she think there are methodological terrorists running around?” he writes. “I can’t be sure about the latter point because she declines to say who these terrorists are or point to any specific acts of terror. Her article provides exactly zero evidence but instead gives some uncheckable half-anecdotes.”
Gelman is not a psychologist himself, but he’s done significant work to identify papers in the field with conclusions premised on faulty or weak statistics, and he is well-respected among the researchers driving the replication debate.
As for who Susan Fiske is, he points to a number of published papers published during her tenure as editor that have fallen apart under close statistical scrutiny. He goes on to cite a paper she wrote that turned out to have a major statistical error and “impossible” conclusions.
In a sense, Gelman confirms Fiske’s claims here. His blog post is personal, attacking not only Fiske’s claims but her authority to make them. As a journal editor and as a researcher, he writes, Fiske has tolerated the “dead paradigm” of bad, invalid statistics propping up shocking results – and refused to retract her conclusions even when an error is found in her methods.
“It’s the paradigm of the theory,” Gelman writes, “that in the words of sociologist Jeremy Freese, is ‘more vampirical than empirical – unable to be killed by mere data.'”
The reason Fiske takes such a harsh stance, he suggests, is that it’s simply too difficult and painful for an established scientist to admit problems with methodologies that have defined her career.
“She’s seeing her professional world collapsing,” he writes. “It’s scary, and it’s gotta be a lot easier for her to blame some unnamed ‘terrorists’ than to confront the gaps in her own understanding of research methods.”
He adds that while Fiske worries about her established colleagues who work under the “dead paradigm,” less-established, more statistics-driven scientists lose out on opportunities to publish their more verifiable work.
But the core of Gelman’s argument is that the new spirit of online debate is constructive and good for psychology, as well as science writ large:
The thing that saddens me is Fiske’s characterization of critics as “adversaries.” I’m not an adversary of psychological science! I’m not even an adversary of low-quality psychological science…. What I am an adversary of, is people not admitting error and studiously looking away from mistakes that have been pointed out to them…
Let me conclude with a key disagreement I have with Fiske. She prefers moderated forums where criticism is done in private. I prefer open discussion… Hence I am posting this on our blog, where anyone has an opportunity to respond… This is open discussion, it’s the opposite of terrorism. And I think it’s pretty ridiculous that I even have to say such a thing which is so obvious.
That is to say, the free flow of criticism is good for science, and that those who want to do good research had better get on board.
A turning point?
As fascinating as it is to peer into a public flash of personal drama among elite research scientists, the real reason this debate is so important is that it comes at a critical moment in the history of psychology.
A friend of mine who has authored several papers on cognitive psychology recently put it to me that psychology is largely in its “pre-Newtonian” era. That is, it’s so young that in many sub-fields no one has yet come along and set down a series of solid laws on top of which a sure-footed body of work can stand.
Without its own versions of An object in motion tends to stay in motion, the science of mind remains in a dark, fluid state where bad research seems to have prospered. The problem’s not universal, but it seems endemic.
Fiske’s letter could go down in history as part of the painful emergence of a new, more rigorous era in psychological science. In the long term, it’s hard to imagine weak statistical work flourishing under the sustained onslaught of statcheck and its adherents. That would, in all likelihood, be a good thing. But the infighting and collateral damage along the way will probably not be fun for anyone involved.
Fiske was unavailable for an interview before this article’s publication deadline. We’ll update this post with her comment if we hear back.