There is widespread support, both among laypeople and researchers, for the system called “peer-review” as a way of certifying the quality of published scientific papers. This must be one of the dumbest ideas that the scientific community has ever produced.
Having your peers discuss your work is great, but that's not what “peer-review” is about. For readers not familiar with the process, it means that when a paper is submitted to a journal, the journal editors will send it to one or two other researchers (the peers) asking them for their opinion about the paper. The reports and the identities of the reviewers remain confidential, and the editors still decide whether or not to publish the paper.
There was nothing wrong with this idea in the first place. The problem is the ridiculous weight assigned to the process. The concept has reached the general public, and more than once I have been lectured on it by non-academics who seem to think that peer-review (or the lack of it) is how we know that homeopathy doesn't work. And within academia, if you apply for a job, a promotion or a research grant, publications that aren't "peer-reviewed" are systematically dismissed.
"Peer-review" is now regarded as an integral part of the scientific method, alongside with diametrically opposite ideas such as openness and skepticism towards unverifiable claims.
The fact that the reviewers remain anonymous is often cited as a reason to trust them, because it means they didn't have to be afraid to criticize the paper. An interesting comparison (that can't be taken too far of course) is the use of anonymous testimony in courts, which has been discussed recently in Sweden. Here the traditional view is the opposite, that a testimony can't be trusted if it's anonymous.
But here is the paradox: If you are suspicious of the quality of a paper to the point that you doubt it’s good even when you are told that it has an author, then how can you possibly become convinced when you get the additional information that it was “peer-reviewed”? This to me makes no sense.
It’s like maintaining a policy of never buying a car unless the salesperson says they asked a person on the street what they thought about it. You don't care who that person were or what they actually said, and you have no means of verifying it anyway, but you maintain a picture of that person as unbiased and therefore trustworthy, and you absolutely insist that the salesperson must say that they asked somebody.
To me it makes more sense just to decide that a Volvo is probably ok.
It’s true that an author might be biased towards thinking that their own paper is great. But at least they put their name on it and gambled a chunk of their reputation. The reviewer is an anonymous person with no incentive to do a good job.
I could go on about how the process is slow and inefficient, and how “peer-review” is used as an excuse for throwing away tax payers’ money on ridiculously expensive journals. By the way, Michael Nielsen has written some interesting stuff about both pros and cons of the system. But the point I’d like to make here is that it’s dangerous to promote “peer-review” as a remedy against fake news and bogus science. I see memes like “What to we want? Evidence-based science! When do we want it? After peer-review!” posted on the internet, and I see claims that only a hundred or so peer-reviewed papers (out of many thousands) cast doubt on anthropogenic global warming.
This makes me think of Goodhart's law: When a measure becomes a target, it ceases to be a good measure. Peer-review is already unreliable, but if we keep promoting it as a gold standard of science, it will also become corrupt, because it's too easy to fake.
We must realize that the fact that the reviewers are anonymous and their reports confidential means we have no way of verifying that those reports even existed, let alone that they endorsed the results. Again I could go on about papers being published without a report or despite reviewers being extremely negative.
But just imagine, out of the many weird things that might happen, that Donald Trump would start his own journal of climate science, claiming it was peer-reviewed. If you insisted for years that climate change denial doesn't hold under peer-review, then what would you do?
The problem is that if you were to question the claim that “We’ve got the greatest people folks, reviewers and really they’re fabulous, and they have the best reviews, believe me.”, then you would be applying double standards. Not good if you claim to be defending the scientific method. You never asked for the reports or the names of the reviewers from any other journal.
And you don’t even need any money in order to start a journal. Heck, I could start my own journal on this blog, claiming it’s peer-reviewed. Again if you make fun of me, suggesting that my peer-review process doesn’t adhere to scientific standards, then the joke is on you, because you just fell victim to confirmation bias.
So let me introduce the Bulletin Of Genuine Unbiased Science (BOGUS), and the special issue on theological number theory.
Paper 0. Proof of the existence of a Largest Prime Number
Abstract: We establish the existence of the Largest Prime Number.
Theorem 0: The Largest Prime Number exists.
Proof: This is derived as a Corollary of the following Lemma.
Lemma 1: Everything exists.
Proof: No counterexample can exist.
No comments:
Post a Comment