Did You Peer At My Article? Facts About Peer Review

 

Go into any family court where social science is being put in evidence,  and you will hear peer review referenced. “Are the papers on your CV in [portentuous pause…] peer reviewed journals?” “You are aware, are you not, that the work you are criticizing was [pause] peer reviewed?” It sounds as if peer review is an assay for gold content rather than what it is—part of the process by which articles are accepted for publications in professional journals.

What is peer review and how is it done? Is it a guarantee that a published article is accurate? This post will attempt to answer all those questions.

When an author submits a paper to be considered for publication in a professional journal, an editor is usually the first person to look at the submission. The editor may decide quickly that a submitted paper is not acceptable for the journal. This might be because the topic is not of much interest to the journal’s usual readers/ There can be many other reasons too. I have had papers rejected because, in one case, the information in the paper was derived from journalistic sources, in another case, because my paper commented on another paper that had been published some time previously, and in a recent case, because the editor was afraid of a lawsuit against the journal if the paper was published there.

If the editor does not quickly reject a paper, he or she will seek peer reviewers who will give opinions about the paper and suggestions for improvements. The reviewers may be members of the editorial board or authors who have published work on similar topics. They receive an anonymous version of the article and their names are also kept secret from the article author. It is not necessarily easy to find two or three reviewers for a paper. Reviewing can be a lot of work, and most people qualified to review an article are also busy with other professional work. (An academic book editor recently mentioned to me that book publishers are having a terrible time getting reviewers for books in development, too.)

Some journals ask authors to suggest some reviewers and even to state whether there are any people they do not want to review their work. I have no idea how those suggestions are actually used.

When reviewers have been found they receive the anonymous manuscript to read. (Authors are even asked to hide citations of their own work by using the word Author instead of their names, but it seems likely that reviewers can sometimes guess whose work they are reading—and some authors can probably guess who their reviewers are.) They reviewers often also receive a list of questions that they should answer as part of their review. These questions, and the focus of the review, are different for different journals. For example, psychology journals are concerned about research design and statistical analysis, whereas journals focused on family law are more likely to focus on legal and practice issues.

Reviewers are often asked to return their completed reviews within a month. When I am a reviewer I usually manage in that time frame, but not everyone does. Between the search for reviewers and the actual review process, peer review may take a year or more. I recently withdrew an article submitted to a journal when it had been a year without any reviews or decisions.

When the reviews have been submitted, an editor will go through them and decide first whether the article should be rejected or accepted as it stands (not common). If the editor decides, as is more usual, to accept the paper if revisions are made in line with the reviewers’ recommendations, he or she may simply send the decision and the reviews to the author, or may summarize the reviewers’ concerns. The editor may add his or her own suggestions for revisions. One of the points the author is asked to make is a list of the weaknesses of the research or reasoning on which the paper is based. The author is given a time limit for resubmitting the revised paper.

If and when the revised paper is resubmitted, the author is also asked to submit a point by point description of the ways the reviewers’ suggestions have been complied with. An editor will review the resubmitted material and decide whether to accept the paper at that point, or to send for another round of reviews, or to reject.

So, these are the steps involved in peer review. They involve many steps and opportunities for human error. Articles may be peer reviewed and yet contain problems that are only noticed later. They may also be correct on the points considered by the reviewers but include misinformation about material that the reviewers did not understand or attend to. Corrections on those points may not occur for years, or ever—or they may be brought up by readers who complain or write rebuttal papers that may or may not be published.

When serious errors are apparent in a published paper, the journal may retract the paper. Depending on when retraction occurs, the paper may not appear at all or may appear with a stamp saying “retracted” on it. Unfortunately, the nature of the Internet is that once such a paper has been posted, it never goes away, and most readers will not know it has been retracted. The retracted paper may even be cited in new papers by other authors. An article by Jeffrey Brainard in Science (1 July 2022) describes unsuccessful efforts to get journal editors to flag citations of now-retracted papers.

The upshot of all this is that peer review is the best thing we’ve got for assuring the accuracy of published articles. It requires great efforts from many people—but it is far from perfect. The best thing readers can do to insure accurate understanding is to be educated, or educate themselves, on the basic issues found in journal articles they want to read and use. In addition, when peer-reviewed articles are used in court, it is critical to remember that research findings are stated in probabilities, not in the simple terms used in law everywhere except Scotland (where “not proven” is an option). Applying the conclusions of journal articles to single cases is a challenge that requires us to think hard even when we would rather have the answer handed to us.