Go into any family court where social science is being
put in evidence, and you will hear peer
review referenced. “Are the papers on your CV in [portentuous pause…] peer
reviewed journals?” “You are aware, are you not, that the work you are
criticizing was [pause] peer reviewed?” It sounds as if peer review is
an assay for gold content rather than what it is—part of the process by which
articles are accepted for publications in professional journals.
What is peer review and how is it done? Is it a
guarantee that a published article is accurate? This post will attempt to
answer all those questions.
When an author submits a paper to be considered for
publication in a professional journal, an editor is usually the first person to
look at the submission. The editor may decide quickly that a submitted paper is
not acceptable for the journal. This might be because the topic is not of much
interest to the journal’s usual readers/ There can be many other reasons too. I
have had papers rejected because, in one case, the information in the paper was
derived from journalistic sources, in another case, because my paper commented
on another paper that had been published some time previously, and in a recent
case, because the editor was afraid of a lawsuit against the journal if the
paper was published there.
If the editor does not quickly reject a paper, he or
she will seek peer reviewers who will give opinions about the paper and
suggestions for improvements. The reviewers may be members of the editorial
board or authors who have published work on similar topics. They receive an
anonymous version of the article and their names are also kept secret from the
article author. It is not necessarily easy to find two or three reviewers for a
paper. Reviewing can be a lot of work, and most people qualified to review an
article are also busy with other professional work. (An academic book editor
recently mentioned to me that book publishers are having a terrible time
getting reviewers for books in development, too.)
Some journals ask authors to suggest some reviewers
and even to state whether there are any people they do not want to review their
work. I have no idea how those suggestions are actually used.
When reviewers have been found they receive the
anonymous manuscript to read. (Authors are even asked to hide citations of
their own work by using the word Author instead of their names, but it seems
likely that reviewers can sometimes guess whose work they are reading—and some
authors can probably guess who their reviewers are.) They reviewers often also
receive a list of questions that they should answer as part of their review.
These questions, and the focus of the review, are different for different
journals. For example, psychology journals are concerned about research design
and statistical analysis, whereas journals focused on family law are more
likely to focus on legal and practice issues.
Reviewers are often asked to return their completed
reviews within a month. When I am a reviewer I usually manage in that time
frame, but not everyone does. Between the search for reviewers and the actual
review process, peer review may take a year or more. I recently withdrew an
article submitted to a journal when it had been a year without any reviews or
decisions.
When the reviews have been submitted, an editor will
go through them and decide first whether the article should be rejected or accepted
as it stands (not common). If the editor decides, as is more usual, to accept
the paper if revisions are made in line with the reviewers’ recommendations, he
or she may simply send the decision and the reviews to the author, or may
summarize the reviewers’ concerns. The editor may add his or her own
suggestions for revisions. One of the points the author is asked to make is a
list of the weaknesses of the research or reasoning on which the paper is based.
The author is given a time limit for resubmitting the revised paper.
If and when the revised paper is resubmitted, the
author is also asked to submit a point by point description of the ways the
reviewers’ suggestions have been complied with. An editor will review the
resubmitted material and decide whether to accept the paper at that point, or
to send for another round of reviews, or to reject.
So, these are the steps involved in peer review. They
involve many steps and opportunities for human error. Articles may be peer
reviewed and yet contain problems that are only noticed later. They may also be
correct on the points considered by the reviewers but include misinformation
about material that the reviewers did not understand or attend to. Corrections
on those points may not occur for years, or ever—or they may be brought up by
readers who complain or write rebuttal papers that may or may not be published.
When serious errors are apparent in a published paper,
the journal may retract the paper. Depending on when retraction occurs, the
paper may not appear at all or may appear with a stamp saying “retracted” on
it. Unfortunately, the nature of the Internet is that once such a paper has
been posted, it never goes away, and most readers will not know it has been
retracted. The retracted paper may even be cited in new papers by other authors.
An article by Jeffrey Brainard in Science (1 July 2022) describes unsuccessful
efforts to get journal editors to flag citations of now-retracted papers.
The upshot of all this is that peer review is the best
thing we’ve got for assuring the accuracy of published articles. It requires
great efforts from many people—but it is far from perfect. The best thing readers
can do to insure accurate understanding is to be educated, or educate
themselves, on the basic issues found in journal articles they want to read and
use. In addition, when peer-reviewed articles are used in court, it is critical
to remember that research findings are stated in probabilities, not in the
simple terms used in law everywhere except Scotland (where “not proven” is an
option). Applying the conclusions of journal articles to single cases is a
challenge that requires us to think hard even when we would rather have the
answer handed to us.