Evidence-Based Treatment: What Is It?

I’ve had a number of annoyed reader comments lately, claiming that certain mental health interventions were evidence-based, and I should stop saying that they weren’t. I’ve been in various brouhahas in professional journals, too, when authors claimed that they were writing about evidence-based treatments (EBTs), and I (and other people too) pointed out that they were not.

Why is everyone so eager to say that they are using an EBT? First, there is considerable cachet nowadays to be gained by saying you have an EBT. Many professional organizations recommend EBTs as the first choice among psychological treatments, and in the interest of accountability, many funding sources require that EBTs be used if available for the needed work.  Second, most people don’t know what the technical term “evidence-based” means, so it’s not too hard to convince them that a treatment is an EBT when it is not.

Everybody knows what “evidence” is in the everyday and the legal senses. It’s information that comes from people’s direct observations or from expert interpretations of indirect factors--  like DNA on underwear or contacts on cellphones. If I tell you what I experienced during a mental health intervention, and whether it made me feel better or not, that’s evidence. But it’s not the “evidence” in “evidence-based treatment”.

“Evidence-based treatment” is a technical term, a “term of art”, or “jargon” if you like that better. It first came into use in the 1990s, when the evidence-based medicine movement began to discuss definitions of the kind of evidence needed to give acceptable support of the effectiveness and safety of a treatment. Soon afterward, psychologists and others began to discuss the idea of levels of evidence--  that the significance of supportive information depended on how the information was gathered. An anecdote or testimonial, for instance, provides a very low level of evidence, and  treatments should not be chosen on the basis of that kind of evidence. To be called EBTs, treatments must have been supported by two independently-done randomized controlled trials; the studies must also meet other requirements such as presenting measures of intervention fidelity (showing that the treatment was done the same way each time). In cases where a treatment cannot be randomized, clinical controlled trials with many restrictions can be used. If a study just looks at people’s conditions before and after treatment, that treatment can’t be said to be an EBT. There has to be a comparison (control) that takes into account the fact that people’s conditions may change spontaneously or with maturation, and it has to be possible to tell how much change occurred that way and how much was caused by a treatment. In addition to these requirements, nowadays there is increasing pressure to include in research reports any evidence that a treatment can be associated with harm, and EBTs need to be reported in ways that allow both potential benefits and risks to be calculated.

Unfortunately, as EBTs have been seen as more and more desirable, the term “evidence-based” has been thrown around ever more loosely. Sometimes this has been done by unethical practitioners who want to increase their business success and know that interesting anecdotes or testimonials will get people’s attention and interest. But sometimes it has been done, I think almost inadvertently, by organizations that aim to provide lists of EBTs for the information of both practitioners and the public.

Such organizations present lists of treatments, but the material must in many cases be read quite carefully before it becomes clear whether a listed treatment is or is not actually to be considered an EBT. Let’s look at two of these--  the National Registry of Evidence-based Programs and Practices; NREPP, www.samhsa.gov/data/evidence-based-programs-nrepp) and the California Evidence-based Clearinghouse for Child Welfare (CEBC; www.cebc4cw.org). Each of these uses a name suggesting that programs listed there should be expected to be evidence-based.

However, NREPP includes the New Age “tapping treatment” Thought Field Therapy on its list, in spite of clear evidence that this method is ineffective. NREPP lists 205 treatments that are primarily for children and adolescents, and mentions possible adverse events for only ten of them. Until 2015, NREPP used a rating and report method that made it easy for readers to assess adverse events and design problems, and it could be calculated that when design problems were assessed, the average rating for handling confounded variables was only 2.6 out of a possible 4.0. A new rating method (which is supposed to be applied gradually to all old reports) makes these and other aspects of studies much more difficult to see.

CEBC lists very few programs that are not aimed at children and adolescents. The website rates treatments from 1 (evidence-based by the definition given earlier) to 5 (concerning methods). But it also classifies some listed treatments as “non-responders” (when proponents did not provide requested material) or as Not Rated (when the material available was not sufficient for a numerical rating). Of 286 programs for children and adolescents, 26 were non-responders and 77 of the “evidence-based” treatments listed were in fact Not Rated. Only 21 of the listed programs were rated as 1, and none whatever were rated 5 (concerning), even though the list included Corrective Attachment Therapy and a “camp” managed by Nancy Thomas--  both associated with adverse outcomes for children.

“Evidence-based treatment” mustn’t be confused with the kind of evidence that we use for decisions in everyday life. The term has a very specific and important meaning--  even more important now that funding sources and third-party payers may reject anything that does not meet the definition of EBT. Unfortunately, even websites that were intended to help the public deal with understanding EBTs are not doing their jobs well, because treatments appear on their lists when they should not. Teachers of introductory psychology classes, listen up—you can help by making this issue a point for your students to understand!