If it's peer reviewed, journalists won't question it
Does the media lose its sense of balance when dealing with peer-reviewed studies? Apparently, some won't even ask mildly skeptical questions, even on politically-sensitive issues. One newspaper editor says,
"We are dealing with a peer-reviewed journal study, and I don't feel at all comfortable going beyond what they are publishing. That is not our role."
This is on a study about marijuana smoking possible causing gum disease. If even the science beat won't question academic studies, what hope is there that sports editors will?
Labels: academics, peer review
posted by Phil Birnbaum @ 2/12/2008 09:18:00 PM
9 comments
9 Comments:
Beware the dangers of "second hand quote!"
Your direct link is to an article in Reason magazine by Jacob Sullum, and you faithfully reproduce his edited version of the quotation. Unfortunately, Sullum does not follow proper quoting practice, both distorting the meaning and hiding the distortion.
Follow Sullum's first link to read that quote in fuller context, which is a reply by the unnamed editor for an unnamed paper (never identified as a "science" editor, as you infer from Sullum's headline, or as an "American" editor, as Sullum says) to a communication by Bruce Mirken [identified as communications director for the Marijuana Policy Project].
Mirken's version of the quote is "As for the rest of your concerns, we are dealing with a peer-reviewed journal study, and I don't feel at all comfortable going beyond what they are publishing. That is not our role."
The editor's reluctance to go "beyond what they are publishing" in the peer reviewed journal [Journal of the American Medical Association, in this case], is not necessarily general, in that they would never be skeptical of a peer reviewed article, or never provide necessary "balance". The editor appears to be responding to specific criticisms by Mirken that they should have "gone beyond" in a particular way.
Let's assume Mirken has quoted and contextualized accurately. Then
Mirken apparently objected to the news story's failure to look for possible "confounding factors" left out of the study which would undercut the strength of the study's finding of an association between 'heavy' marijuana use and gum disease. We don't know specifically what Mirken put in his communication to the editor, but he mentions two such factors in his blog column(?) - alcohol use and dental hygiene habits (brushing and flossing).
On the first Mirken may have a point; on the second he appears off base, because the study involved direct inspection of plaque buildup on the subjects' teeth, which would seem to be more reliable data than any self-reporting about hygiene practices.
Without seeing the original news article, Mirken's full original communication to the editor, and the editor's full reply, I hesitate to draw any conclusion whatsoever about the policy of this particular editor regarding peer reviewed literature. Certainly I am not willing to generalize to "the media" or "the science beat."
Fair comment; I should have read the original more carefully. I will fix the incorrect reference to it being a science editor.
However: on rereading, Mirken does say specifically that the quote in question was in response to a specific question by Mirken asking why the paper didn't address those specific points.
And whether Mirken's criticisms are correct or not do not factor into this particular argument. The editor didn't say, "we didn't use your points because we felt they weren't valid." He or she said, "we didn't present a counterargument because this is a peer-reviewed study."
Unless Mirken is mistaken or lying, it does seem to me that Reason's point is valid.
I think you're asking a lot here. Certainly journalists are paid to be skeptical. But peer-refereeing exists for a reason, which is to make sure that the research underlying the pirec is sound (meets generally-accepted scientific practices).
In this particular case, it sounds like Sullum has a point of view and is looking for a reason to dispute findings that he does not like. Vague criticisms of a study don't cut it. (For example, when you disagree with the results of a peer-reviewed study, you present your reasons for doing so, often based on research you've done, or that someone else has done...I'll admit that I thin some of your ciriticisms could use a little peer review, but, then, your commenters often provide you with that.) Disliking the results doesn't cut it.
Asking newspapers to, in effect, provide additional reviews of technical scientific papers really is too much to ask.
I agree with you that asking newspapers to *review* academic papers is too much to ask (except in the case of a serious science report). However, when the study touches on a subject of some controversy, it's a good idea to get another view, just as you would when it's an opinion, rather than peer review.
For instance: Policitian X calls a press conference to argue that smoking should be banned in restaurants because secondhand smoke kills hundreds of people a year. For balance, the reporter should get a reaction from a restaurant trade association, or a smokers' rights group. Agreed?
Now, academic researcher Y calls a press conference to publicize his study that shows that secondhand smoke in restaurants kills hundreds of people a year. Shouldn't the reporter get a reaction from a restaurant trade association, or a smokers' rights group, after they have read the study and can comment on it?
The idea that you don't have to question or balance anything that's peer-reviewed is bad journalism. Peer review is not a substitute for normal journalistic skepticism.
But perhaps I'm overreacting a bit ... thinking about it more, I'm not sure the unnamed editor really means what he says. If the study's results were politically incorrect -- say, a finding that smoking marijuana increases your lifespan, or that whites have a higher IQ than blacks -- you can be absolutely sure that the reporter would have found someone to question the results.
So, maybe the marijuana guy was just getting the brush-off, and "peer reviewed" was a convenient excuse.
If anyone is interested in making an independent judgment of Mirken's claims about what efforts the unnamed newspaper should have made, here is the underlying article at the root of his exchange with the editor. [Not that I think we can safely infer exactly what journalistic efforts Mirken asked for.]
Mirken complains in general about sensational headlines and uncritical reporting (without ever bringing political sensitivity or "political correctness" into it at all).
Phil, as to your #5, Mirken himself was not asking for "balance", or quotation of non-scientific opinions, so I don't think his conception of good journalism is the same as yours. Nor do I share your view. The news here was that some scientists made a claim. Who, what, when, where, why. Accurately reporting the claim is all that good journalism requires. Balance may come in other ways, but not by automatically embedding a counterpoint in every article. I have no idea why you suggest "serious science reports" should be held to a different standard than other types of news articles.
And in the interest of brevity, I'll just say I am not persuaded by your characterizations in #3.
Hi, Joe,
"Balance may come in other ways, but not by automatically embedding a counterpoint in every article."
Agreed. My objection is only to the viewpoint that if something is peer-reviewed, it need not be questioned at all. I also agree with you that if the article was a short one, simply outlining what the study said, an opposing view might not be required.
What I'm saying is that if the editor truly implied that the paper doesn't feel a need to cast a critical eye because the study was peer reviewed, that's wrong.
"Accurately reporting the claim is all that good journalism requires."
Are you sure you mean this? What kind of claim do you mean here -- just peer-reviewed claims?
Oh ... and whether Mirken's claims are correct or incorrect has no bearing.
Certainly if you argue that the editor has no obligation to read and evaluate the study, it has no basis to evaluate Mirken's argument against it.
For all I know, Mirken might be full of it. He might be mistaken, or lying, or just shading his position in favor of his pro-marijuana stance. Regardless, if he has criticism of the study, or anything the authors are quoted as saying in the article, the media should consider reporting that.
My only point is: "the study is peer-reviewed" is not, in and of itself, justification for ignoring legitimate criticism of it.
Post a Comment
<< Home