In response to a recent post, an anonymous commenter wrote that
In the age of the Discovery Institute there are some good reasons to focus on peer reviewed research as a way of excluding quacks. It's a way of saying that this research has been vetted.
That said, when I read comments like the one above, I think the time has come to push back, and point out that peer review is not the arbiter of truth. Truth is the arbiter of truth, and peer review is merely a flawed tool we use to help get there.
Peer reviewers don't check to make sure the results are true. Peer reviewers do not typically replicate the experiment in question. They do not check the math. Most of what they do is check that the arguments are reasonable and that the experiment(s) were well designed. Peer reviewers do not necessarily even have to agree with a paper they accept. They may simply think the data are compelling and the arguments are worth hearing, even if they may be wrong.
Thus, peer review does a reasonable job of weeding out quacks. Luckily, most scientists are not quacks, so what does it do for the rest of us? I'm not sure, but I think a partial answer is that two minds are better than one. Reviewers often notice things that the authors missed -- not because the authors weren't smart, but because research is damn complicated and you can never think of everything.
Typically what happens, at least in psychology, is that the reviewers suggest additional analyses or additional experiments that would make the paper stronger. Based on those comments, the authors may run new experiments then revise the paper and resubmit. Peer reviewers, in this sense, aren't so much vetters or fact-checkers as editors. Peer review is a way of improving -- not perfecting -- an article.
So is it scientific misconduct to refer to unpublished work? I don't think so. It is dangerous, though, because there are good reasons (above) to be more confident of something that has gone through peer review. It is a bit impolite to refer to something that has not been published, because then your audience can't go look at it themselves. And that's the main point. Peer reviewers are not the judges of truth, but all of us are on the jury.
It would be scientific misconduct ... to make statements based on someone else's unpublished work ... Scientific results don't exist until they have been peer reviewed and published.Peer review has become the gold standard of the scientific community. Bring up a scientific finding, and the first thing you may be asked is, "Ah, well, is this peer reviewed?" (For those who don't know, peer review means that, before the journal will publish a paper, one or more other scientists who study similar topics). There is now even a popular blog aggregater that focuses exclusively on blogging about peer reviewed research.
In the age of the Discovery Institute there are some good reasons to focus on peer reviewed research as a way of excluding quacks. It's a way of saying that this research has been vetted.
That said, when I read comments like the one above, I think the time has come to push back, and point out that peer review is not the arbiter of truth. Truth is the arbiter of truth, and peer review is merely a flawed tool we use to help get there.
Peer reviewers don't check to make sure the results are true. Peer reviewers do not typically replicate the experiment in question. They do not check the math. Most of what they do is check that the arguments are reasonable and that the experiment(s) were well designed. Peer reviewers do not necessarily even have to agree with a paper they accept. They may simply think the data are compelling and the arguments are worth hearing, even if they may be wrong.
Thus, peer review does a reasonable job of weeding out quacks. Luckily, most scientists are not quacks, so what does it do for the rest of us? I'm not sure, but I think a partial answer is that two minds are better than one. Reviewers often notice things that the authors missed -- not because the authors weren't smart, but because research is damn complicated and you can never think of everything.
Typically what happens, at least in psychology, is that the reviewers suggest additional analyses or additional experiments that would make the paper stronger. Based on those comments, the authors may run new experiments then revise the paper and resubmit. Peer reviewers, in this sense, aren't so much vetters or fact-checkers as editors. Peer review is a way of improving -- not perfecting -- an article.
So is it scientific misconduct to refer to unpublished work? I don't think so. It is dangerous, though, because there are good reasons (above) to be more confident of something that has gone through peer review. It is a bit impolite to refer to something that has not been published, because then your audience can't go look at it themselves. And that's the main point. Peer reviewers are not the judges of truth, but all of us are on the jury.
No comments:
Post a Comment