Top academic journals won't publish rebuttal replications
Recently, a top academic psychology journal, JPSP, decided to publish a study purporting to show the existence of ESP. An author had submitted a paper, which showed statistical significance at 2.5 SD, and the journal argued that it wasn't appropriate to reject the paper just because the effect was precognition rather than something more mainstream.
So the paper was accepted. The decision to publish was controversial, as you might expect. Almost immediately, several researchers replicated the experiment, and found no effect whatsoever. They submitted rebuttal papers to JPSP.
And JPSP refused to publish them! Not only JPSP, but "the other high-end psychology journals" (according to this Carl Shulman post from "Less Wrong", which I am paraphrasing here) also refused.
Their explanation: they don't publish straight replications.
So, let's get this straight. Someone writes a study that makes extremely bold claims with weak evidence. The journal decides to publish it. But when other researchers almost immediately rebut it in the strongest possible way -- by replicating the experiment exactly -- the journals decide they're not interested.
Shulman writes,
From the journals' point of view, this (common) policy makes sense: bold new claims will tend to be cited more and raise journal status (which depends on citations per article), even though this means most of the 'discoveries' they publish will be false despite their p-values. However, this means that overall the journals are giving career incentives for scientists to massage and mine their data for bogus results, but not to challenge bogus results by others. Alas.
As far as I'm concerned -- and, indeed, as far as the scientific method is concerned -- it isn't science unless you expose your findings to confirmation and challenge. By Shulman's argument, it looks like academia has sacrificed the pursuit of science to the pursuit of status-seeking, where journals want to be interesting and professors want to avoid being proven wrong.
Labels: academics
posted by Phil Birnbaum @ 8/13/2011 10:16:00 AM
3 comments
3 Comments:
Let me start by saying, I 100% agree that it is a shame and, I think, wrong for JPSP to not publish rebuttals, especially if there are a number of them from different labs. But it is not true that journals will not publish any rebuttals or that academia has moved toward some manner of sensationalism.
I was able to read at least three rebuttals to the Bem paper before it was even formally published, and to my knowledge they were all published (if not in JPSP, although one was). A journal that focused more in my field has run a number of 'topic' issues, where the entire issue is dedicated to different perspectives on some topic. You can be certain that the papers all don't agree with each other. I have a number of articles that all have "A reply to___" in the title. It's fair to say that many of these rebuttals or replications aren't in the absolute top journals but they are easily findable, either by the same search that would get you an article like Bem's or by using the handy 'cited by' link present in most search engines (Google Scholar or PubMed just to name two). Rebuttals to an iffy paper will not be hard to find.
The counter to the argument that the system encourages sensational claims is to say that what it really encourages is groundbreaking work. When making a new experiment my adviser has always told me to have a piece that you know will work, or replicate existing work, and then add something new. That is how science progresses, by incrementally establishing findings and testing how they might be expanded or where they might break down. A straight replication is valuable, but not as valuable as showing that the same effect also happens under these other conditions, or it fails to happen. Good science articles should contain something new.
A final point to make is that I think it's a mistake to blame JPSP or the academic system in general for publishing Bem's paper. You can actually view it as a rebuttal; it's a rebuttal to the common view that ESP doesn't exist, and all the papers in the past that have failed to find evidence for ESP. If researchers never published results that conflicted with other data or common knowledge, science will truly have failed. Just because many of us (including myself) disagree with Bem's paper doesn't mean that it couldn't be true; many people used to disagree with the idea that the Earth wasn't the center of the universe. Most of the complaints about Bem's article related to the data analysis, but Bem met the same standards of evidence that people had met in the past when showing that ESP didn't exist. There was nothing 'wrong' with his paper; it would have been published anywhere. I saw the whole thing as a call for better statistical knowledge rather than an indictment of the publishing system or academia.
That's pretty crazy. Why have a journal at all if you are not going to accept the fact that the stuff you put in it should be falsifiable?
Alex's comment was marked as spam. It has now been freed.
Post a Comment
<< Home