Saturday, September 03, 2011

Academic editor resigns after publishing flawed study

The editor of an academic journal has resigned after publishing a study that turned out to be flawed.

There's more to it than that, of course ... it's mostly an issue of political correctness, rather than a scholarly one. The study in question was by a politically incorrect author, with a politically incorrect conclusion. The paper, it turns out, was skeptical of climate change, and there are accusations that the peer reviewers who gave it their blessing were also known skeptics.

Still, it's interesting to see how the reaction pretends that's not an issue. The resigning editor, Wolfgang Wagner, wrote:

"[The paper was] "fundamentally flawed and therefore wrongly accepted by the journal ... As the case presents itself now, the [peer review] editorial team unintentionally selected three reviewers who probably share some climate sceptic notions of the authors …the problem I see with the paper by [authors] Spencer and Braswell is not that it declared a minority view (which was later unfortunately much exaggerated by the public media) but that it essentially ignored the scientific arguments of its opponents."

So, let me get the implications straight.

1. It is a very serious matter if a flawed paper is accepted by a journal.
2. If the peer reviewers agree with the author on a related scientific theory prior to the paper being published, you should find other peer reviewers.
3. If a paper ignores the scientific arguments of its opponents, it should not be published.

Can these people possibly be serious?

1. Flawed papers are accepted by journals ALL THE TIME. The sabermetric community has revealed the flaws in many, many academic studies, and no editor has resigned. Indeed, on several occasions, we have revealed problems with studies when they're still in the "working paper" stage, and they get published anyway.

If an editor had to resign every time a flawed paper got published, no editor would last longer than three months in the job.

2. This only seems to become a principle when the scientific theory in question is politically incorrect. If a journal considers a study *confirming* climate change, do they really go out of their way to find climate skeptics to peer review it? If a psychology journal publishes a study documenting the negative effects of racial bias, do they insist that one of the peer reviewers be a KKK member?

3. Actually, I'm OK with this one. But I can't resist snarping a little bit. Has any editor ever been forced to resign because of the publication of an academic paper on sabermetrics that doesn't know who Bill James is, that barely cites any existing sabermetric research, and that could have been refuted by a sabermetrician in fifteen seconds?

Okay, I'm done snarping. Moving on now.


Have you ever noticed what a big deal it is in academia whenever a study is acknowledged to be flawed? The hands wring, the "mea culpa"s flow, and everyone talks about what could have gone wrong with the process that this was allowed to happen.

It looks good at first, that academia is so concerned about getting it right that they take it so seriously when something is wrong. But, really, it's a veneer, isn't it? They're just trying to signal how serious and ethical they are to people who don't know any better.

More often than not, it doesn't work that way in real life. We've all seen and talked about studies that are obviously flawed, and we've seen examples of academics who deflect the arguments to irrelevant side issues, ignore them completely, or attack our credentials instead of the actual criticism at hand.

Sometimes the community will be a bit more subtle than that ... they won't disown the study explicitly; instead, they'll publish a rebuttal letter, or a study opposing the original. They'll try to position it as a healthy scientific debate between scientists.

But, no matter how flawed the paper, they don't normally demand that the editor resign. And there also seems to be an implicit understanding that you don't pillory the original author, even if the original study was obviously meritless. You just make sure you never cite it favorably, and everyone ignores it and gets on with their lives.

But here, they won't do that. The climate change research community seems to hold the offending author in very low regard. Normally, they'd just ignore the offending author, knowing their peers would extend to them the same courtesy. But it seems like they're just fed up with this Spencer guy. And, perhaps that's for good reason. As one professor said,

"Spencer [one of the co-authors] is well known in the scientific community for publishing high-profile papers that initially dispute global warming and only later are found to be faulty."

Still, this is not a case of academia standing up to defend its strict standards of truth. This seems to be a case of academia having decided that this particular academic is persona non grata on this particular subject, and that they're not going to let him, or his editor, get away with things other academics can.


And you know, this turn of events might have been totally the right thing to do. I don't know this Spencer guy. For all I know, he might be an awful scientist, blinded by his political beliefs, trying to publish bad papers anywhere he can get away with, to cast doubt on the climate change hypothesis. In that case, we might agree that because of his repeated disingenuousness, and the poor quality of his work in the past, his latest study should have been subject to extra scrutiny -- and his editor's failure do that represents a resignable offense. (Again, I don't know Spencer's work at all, so this is entirely hypothetical.)

But if that's the case, say so! Don't say "the editor was fired because the paper was flawed." It makes you academics sound ridiculous, like you consider yourselves more infallible than the pope. It sends the idea that everything that makes it into a journal is invariably 100 percent correct -- because, if it weren't, would the editor still be working here?

If you say, "the editor was fired expected to resign because the paper was flawed," you sound arrogant and silly, not to mention dishonest.

The critics should just tell the truth. They should say, "Look, there's this one guy who's acting like a dork. He's putting together these crappy studies, which have no scientific merit, and he won't do what scientists are supposed to do and look at the data objectively. He's so politically committed to his hypothesis that he doesn't care about making his studies hold together, and he gets everything wrong.

"Now, we're scientists, so we are very open to the idea that our current theories might be wrong, and there are skeptical scientists whose research is valid and whom we respect. But not this guy. His work has been so bad, for so long, that it's incumbent on any editor to double and triple-check his work to make sure he's not doing it again. In that light, when his editor failed to do that, it's akin to negligence. So it's only appopriate that he resign."

That would make sense. But, I guess, to the general public, it doesn't look as good.



At Saturday, September 03, 2011 3:34:00 PM, Anonymous Anonymous said...

You should get over your jealous bitterness toward "academics" (as if academia is homogenous) and (1) let "sabremetrics'" work stand for itself vis-a-vis "academic studies", (2) figure out how to make the sabermetrics community more competitive with academia so it can get the credit you think it deserves (3) get a Ph.D., join the tribe, and show that you can do better.

We get it: Lots of academic research is bad. But so is lots of (most?) saber research. Quality is quality; it doesn't matter where it comes from. Does the media slight "uncredentialed" sabermetricians in favor of academics doing similar work? Probably. Why? The average Ph.D. has (1) has more methodological training than the average sabermetrician, and (2) more colleagues (who are at least equally well trained). Consequently, the chaff is more likely to be separated from the wheat prior to academic publication, even at the working paper stage. You pick on academic research where this process fails, but you're not sampling randomly either, are you?

At Saturday, September 03, 2011 7:27:00 PM, Anonymous matskralc said...

Phil, I think what Anonymous is saying is that this blog post wouldn't make it into a peer-reviewed journal. :(

At Saturday, September 03, 2011 8:57:00 PM, Blogger paulik said...

The editor in chief was not fired or forced to resign, he just took the honorable course. I hope I didn't misunderstand you but it seemed that you were saying that he was.

At Saturday, September 03, 2011 11:37:00 PM, Blogger Phil Birnbaum said...

Paulik: fair enough, I'll fix the wording.

At Monday, September 05, 2011 7:35:00 PM, Anonymous Giles said...

My only knowledge of academic peer review is from my sister who is a published Doctor.

But from what she has told me in the past it is very bad for something to be published in a reputable scientific journal and then turn out to be incorrect.

One of the biggest problems with the way scientific research is done is that you are not allowed to write up and publish your work if your experiment does not work or your line of experimentation turns out to be false.

As a result there ends up with lots of people doing the same research and finding out the same wrong conclusions because there is no way to share what doesn't work.

Now I am not saying that there are not some journals which will publish things that are wrong or that some people can write statistical papers that can be easily disproved but in this case, for this to have been published, was a huge mistake and the editor is acknowledging that.

The fact that this is a political hot topic has only served to make things worse.

At Tuesday, September 06, 2011 12:37:00 PM, Anonymous Anonymous said...

I'd say this study was like doing research on clutch hitting, but only looking at the stats from the winning team. I guess that could be published and "clutch" believers would be excited, but there would be no value. Basically, if something that flawed makes it thru review, it's not a badge of honor to have published it.

At Wednesday, September 07, 2011 2:13:00 PM, Anonymous Adam H said...

Could I get a link or citation on exactly how this study is flawed? Other than going against the consensus...

At Wednesday, September 07, 2011 3:54:00 PM, Blogger Phil Birnbaum said...

Here's something I found that links to other things too.


Post a Comment

<< Home