Peer Review Is Better Than Nothing (But Not Magic)

I recently saw a nice image of a demonstrator with a sign that said: “What do we want? Evidence-based policy. When do we want it? After peer review.”  And, yes, that is what we want—but it doesn’t pay to assume that peer review is the complete answer to understanding whether evidence is reliable.

Plenty of publications are not peer-reviewed. Newspapers and popular magazines are good examples. Articles in these publications may have been requested and paid for by an editor, or chosen by an editor from material submitted by writers. High-quality newspapers and magazines have fact-checkers and run specialized material past knowledgeable people (as well as past their lawyers, who decide whether a charge of defamation will result from publication). Lower-quality journalism may simply use articles because they are sensational and will help sales or because they support an organization’s claims and policies. The same issues apply to Internet materials as to print publications.

Professional journals, however, are supposed to be peer-reviewed, and most of them carry on a front or back page the statement that articles are assessed by peer reviewers. This means that an editor who receives a submitted article will choose two or three people who are thought to be experts in a field (often members of the journal’s editorial board) and will ask them to read and comment on an article, make a recommendation about acceptance or rejection, and list minor or major changes that they think are needed before the article is published. This can be a lot of work and may or may not be done well, depending on the peer reviewers selected. (If the article topic is unusual, it can be hard for an editor to find knowledgeable peer reviewers.)

The idea of peer review is that reviewers are supposed to know the background and previous publications on a topic, but even  though specialists usually have opinions on their specialty, they are expected to be able to take an unbiased approach. The paper is sent to the reviewers without the author’s name on it, but in some specialized fields the reviewers do know who is working on what topic and may be able to guess who wrote the “anonymous” paper. We can hardly expect reviewers to be so pure that they can ignore their own opinions or their guesses about an author’s identity (BTW, authors can sometimes guess reviewers’ names too), but most reviewers do try hard to be objective. The work they do to suggest revisions to a paper helps them do this by making them focus on what would be needed to improve an article.

Peer reviewers are supposed to consider whether an author has paid attention to relevant current literature, whether the design or internal logic of a study supports the conclusion drawn, whether any statistical work was done correctly and whether the number and nature of the participants in an empirical study was sufficient for the purpose, and whether there are ethical issues to be discussed. Reviewers also need to pay attention to the way an article is written and whether it is clear enough that readers will understand it. The reviews that go to the author need to touch on these matters and often include specific details.

Once an author has received a set of peer reviews from an editor, it’s his or her job to revise the paper to meet the recommendations of the reviewers and respond to their critiques—or argue why those changes would be inappropriate. The revised paper is then returned to the editor with a list of reviewers’ comments and how the author has responded to them, and the editor may decide on his or her own to accept or reject the paper as it stands, or may return it to the reviewers for further decisions. (This is one of the reasons it takes a long time for an article to get into print.)

As you can see, this is a very well-intentioned process that can help keep weak or even fraudulent material out of professional journals--- when everything goes according to plan. It’s a better approach than anything else we have at the moment and certainly more desirable than letting every submitted article be published or leaving all decisions to the preferences of a few editors, who can’t possibly be knowledgeable about every area of their discipline. But at the same time, look at the potential flaws that mean that there is no magic in peer review. Mistakes may be made in choosing reviewers, or reviewers with a particular bias may be chosen by some journal editors. Being human, reviewers can also make mistakes of all kinds, fail to pay attention to every point they should consider, or take on too much work so they do not have time to do a good job. They may also fail to completely understand the goals of a journal--  for example, another reviewer and I once bent over backward to make suggestions about how an author should revise a paper with a questionable topic, only to have the editor reject the paper out of hand because of the subject matter and the way it was handled.

Because peer review is a human process, we should never assume that a paper that has been published even after excellent peer review is necessarily True with a capital T. The burden remains with the reader to consider the weight that should be given to a publication. It’s very helpful to know that a few other people have critiqued a paper, and certainly reviewers’ comments can help an author improve a paper a great deal. But there’s no magic truth in peer review – it’s just better than the alternative.

Another thing about this: sometimes editors invite an author to submit a paper. Those invited papers are not peer reviewed, but the authors are free to say that their paper was published in a peer-reviewed journal. This is the case for an article by Forrest Lien currently posted on line for Child and Adolescent Social Work Journal. This invited paper is basically a puff piece and would not have been accepted if it had been submitted in the usual way, but it was invited in order to give Lien a chance to respond to some criticism of his methods. Unfortunately it also gives him a chance to say that there is a peer-reviewed publication recommending Lien’s approach.

Finally, there are presently quite a lot of “predatory journals”. These are outfits that invite submissions but conceal the fact that they are going to charge for publication, refuse to let an author withdraw a paper, and put a collection agency to work if the author does not pay up. Obviously the papers that end up in those publications are not peer-reviewed no matter what the journals say. (I get half a dozen of these journals contacting me every day, and I’m sure this is also true for anyone who has published in a real professional journal.)

The moral of our story is Caveat Lector—let the reader beware! Peer review is good, but the responsibility for evaluating publications remains in our own hands.