Did You Peer At My Article? Facts About Peer Review

 

Go into any family court where social science is being put in evidence,  and you will hear peer review referenced. “Are the papers on your CV in [portentuous pause…] peer reviewed journals?” “You are aware, are you not, that the work you are criticizing was [pause] peer reviewed?” It sounds as if peer review is an assay for gold content rather than what it is—part of the process by which articles are accepted for publications in professional journals.

What is peer review and how is it done? Is it a guarantee that a published article is accurate? This post will attempt to answer all those questions.

When an author submits a paper to be considered for publication in a professional journal, an editor is usually the first person to look at the submission. The editor may decide quickly that a submitted paper is not acceptable for the journal. This might be because the topic is not of much interest to the journal’s usual readers/ There can be many other reasons too. I have had papers rejected because, in one case, the information in the paper was derived from journalistic sources, in another case, because my paper commented on another paper that had been published some time previously, and in a recent case, because the editor was afraid of a lawsuit against the journal if the paper was published there.

If the editor does not quickly reject a paper, he or she will seek peer reviewers who will give opinions about the paper and suggestions for improvements. The reviewers may be members of the editorial board or authors who have published work on similar topics. They receive an anonymous version of the article and their names are also kept secret from the article author. It is not necessarily easy to find two or three reviewers for a paper. Reviewing can be a lot of work, and most people qualified to review an article are also busy with other professional work. (An academic book editor recently mentioned to me that book publishers are having a terrible time getting reviewers for books in development, too.)

Some journals ask authors to suggest some reviewers and even to state whether there are any people they do not want to review their work. I have no idea how those suggestions are actually used.

When reviewers have been found they receive the anonymous manuscript to read. (Authors are even asked to hide citations of their own work by using the word Author instead of their names, but it seems likely that reviewers can sometimes guess whose work they are reading—and some authors can probably guess who their reviewers are.) They reviewers often also receive a list of questions that they should answer as part of their review. These questions, and the focus of the review, are different for different journals. For example, psychology journals are concerned about research design and statistical analysis, whereas journals focused on family law are more likely to focus on legal and practice issues.

Reviewers are often asked to return their completed reviews within a month. When I am a reviewer I usually manage in that time frame, but not everyone does. Between the search for reviewers and the actual review process, peer review may take a year or more. I recently withdrew an article submitted to a journal when it had been a year without any reviews or decisions.

When the reviews have been submitted, an editor will go through them and decide first whether the article should be rejected or accepted as it stands (not common). If the editor decides, as is more usual, to accept the paper if revisions are made in line with the reviewers’ recommendations, he or she may simply send the decision and the reviews to the author, or may summarize the reviewers’ concerns. The editor may add his or her own suggestions for revisions. One of the points the author is asked to make is a list of the weaknesses of the research or reasoning on which the paper is based. The author is given a time limit for resubmitting the revised paper.

If and when the revised paper is resubmitted, the author is also asked to submit a point by point description of the ways the reviewers’ suggestions have been complied with. An editor will review the resubmitted material and decide whether to accept the paper at that point, or to send for another round of reviews, or to reject.

So, these are the steps involved in peer review. They involve many steps and opportunities for human error. Articles may be peer reviewed and yet contain problems that are only noticed later. They may also be correct on the points considered by the reviewers but include misinformation about material that the reviewers did not understand or attend to. Corrections on those points may not occur for years, or ever—or they may be brought up by readers who complain or write rebuttal papers that may or may not be published.

When serious errors are apparent in a published paper, the journal may retract the paper. Depending on when retraction occurs, the paper may not appear at all or may appear with a stamp saying “retracted” on it. Unfortunately, the nature of the Internet is that once such a paper has been posted, it never goes away, and most readers will not know it has been retracted. The retracted paper may even be cited in new papers by other authors. An article by Jeffrey Brainard in Science (1 July 2022) describes unsuccessful efforts to get journal editors to flag citations of now-retracted papers.

The upshot of all this is that peer review is the best thing we’ve got for assuring the accuracy of published articles. It requires great efforts from many people—but it is far from perfect. The best thing readers can do to insure accurate understanding is to be educated, or educate themselves, on the basic issues found in journal articles they want to read and use. In addition, when peer-reviewed articles are used in court, it is critical to remember that research findings are stated in probabilities, not in the simple terms used in law everywhere except Scotland (where “not proven” is an option). Applying the conclusions of journal articles to single cases is a challenge that requires us to think hard even when we would rather have the answer handed to us.

 

Can We Regulate or Ban Potentially Harmful Psychological Treatments?

 

If a psychosocial treatment is powerful enough to change someone’s thoughts, feelings, and behavior for the better, we need to be aware that it would also be possible for the treatment to change matters for the worse. Psychological treatments, like medical practices, can have the potential for harm to the recipient. When the person being treated is a child or adolescent, it’s quite possible that harm to the client can be accompanied by harm to some or all of the rest of the family.

Unlike medical scenarios, where severe physical harm or death can result from some treatments, psychological treatments do not generally cause immediate and visible harm. The harm done by some psychological treatments may be delayed, as would be the case in later family dysfunction or suicidal thinking and actions. The harm may even be indirect, as in the wasting of time and resources a family needs for a range of expenditures like dance lessons for another child or home remodeling to give children their own rooms. The use of potentially harmful, ineffective treatments also delays or makes impossible other treatments that are safe and effective for children and families.

An example of a potentially harmful treatment is Critical Incident Stress Debriefing, CISD, an approach to preventing trauma responses in people exposed to disasters. When CISD was used following 9/11, it became apparent that people who did not receive CISD treatment did better in terms of trauma symptoms than those who did receive it. Psychologists and other mental health professionals were advised not to use CISD, but the Internet continues to carry positive messages and encouragement to use CISD.  

It is not illegal to use CISD, and as the treatment is used primarily for adults, there has never been real pressure for legal prohibition. When children or adolescents are harmed by psychosocial treatments, it is much more likely that there will be pressure to regulate those treatments by legislation that prohibits their use with minors.  But even those regulatory bills are not always passed, or if they are passed do not always speak to the real issues.

In 2001, the state of Colorado passed Candace’s Law, which prohibited active physical restraint as a therapy. This law was instigated by the death of a 10-year-old girl, Candace Newmaker, by suffocation during a psychotherapy purporting to cause her to become emotionally attached to her adoptive mother. The therapists believed that they could accomplish this goal by “rebirthing” treatment, ordinarily a brief, silly but harmless procedure. Candace was wrapped tightly in a sheet and told that she must struggle to get out and be “reborn”, while helpers pressed rhythmically on her body in mimicry of uterine contractions. The child was not able to comply and was asphyxiated. The therapists directing the procedure were convinced that Candace had to be forced into compliance and ignored her pleas that she could not breathe. The Colorado legislation was pushed through because of public horror at the event as well as because of a small number of activists pressing for passage, but some Colorado therapists continued to support “rebirthing” and other methods involving physical restraint. Adoption organizations supported the legislation because many of the children involved in these treatments were adopted.

In 2003, the state of Utah attempted to pass legislation banning the use of “holding therapy”, a treatment involving physical restraint that was related to the “rebirthing” that killed Candace Newmaker. (Candace had also been subjected to holding therapy, whose tenets supported the view that she must be forced to comply.) However, the bill banning holding therapy never came to a vote as opponents ran out the clock on the final day of the legislative session. Mental health professionals and activists strongly supported the bill, but it was opposed by practitioners of holding therapy in the state.

In 2015 and later, more than 20 states passed legislation prohibiting psychologists from using conversion therapies with minors. Conversion therapies are psychological treatments intended to alter same-sex orientations to heterosexual orientations; these treatments involve a range of unconventional methods such as mock-fighting while naked or  using practices related to holding therapy. Conversion therapies have been very distressing to a number of participants and have never been shown to be effective. Conversion therapies remain legal when chosen by adults for themselves or when performed with children and adolescents by members of the clergy. State psychological associations argued strongly in favor of the legal ban, as did LGBT organizations that wielded considerable political power.  A bill introduced in the House of Representatives in 2021 prohibited states from using Medicaid funds for conversion therapy, which was defined as attempts to change sexual orientation in return for monetary compensation.

In 2022, the reauthorization of the Violence Against Women Act included a model law, Kayden’s Law, which states must enact if they are to be eligible for any of the funds appropriated under VAWA. Kayden’s Law is directed at the use of so-called “reunification therapies”, treatments that purport to correct the attitudes of children who display “parental alienation” (resistance to or refusal of contact with one of their divorced parents). Among other potential harms, use of these treatments has sometimes placed children in the hands of abusive parents.The treatments are generally done under court orders that prohibit any contact between the child and the preferred parent for 90 days or longer; the treatments are claimed to create a good relationship between the child and the rejected parent.  The model law states that children are not to be separated from the preferred parent for the purpose of encouraging their relationship with the disfavored parent, a provision that would essentially prohibit most reunification therapies.. California has already reported its bill, SB616, out of committee, while in Massachusetts the bill is still in committee. These bills are bound to be opposed and challenged by a number of  lawyers and mental health professionals who are deeply involved with parental alienation cases in spite of the lack of evidence for the safety and effectiveness of reunification therapies. It will also be opposed by many fathers’ rights groups and by organizations like Family Access Fighting for Children’s Rights that promote the use of the treatments. Parents who have been ordered not to contact their children will also be strongly interested in the legislation, but in many cases those parents are already bankrupt because of the expenses of their legal fights for custody of their children. Psychologists and psychiatrists as professional groups have not taken official positions on this legislation.

When potentially harmful psychological treatments are not regulated by professional organizations, what does it take to ban them by legislation? There is a horrible advantage to having a child die in the treatment and become a “crystallized symbol” or “poster child” for opposition to the practice. Even then, however, there will be those who favor the problematic practice and want it to continue, who consider the death to be a pure accident (or, in Candace Newmaker’s case, who suggest that the child died on purpose in order to distress people). When harms are indirect and not immediate, there are plenty of people who will say they are negligible harms. Successful legislation may depend on activism by those with a personal stake in the game, with financial and other resources, and with the capacity to organize for political purposes. These factors have played a critical role in the legislative banning of conversion therapies. Will they exist or be successful with respect to treatments for parental alienation? That remains to be seen. It is clear from the examples given in this post that there is no point waiting for professional groups to take the first strong positions regulating potentially harmful psychological treatments. But it may happen that when legislation is in progress, some professional groups will feel that they must speak out one way or the other.