I wrote the following paper for a conference about a year ago, and recently a colleague asked whether it was on line anywhere-- so here it is, for those interested. (Since writing this, I have come across cases of confusion about the meaning of the CEBC evaluations and the conflation of "relevance" scores with research scores.)
Internet Registries and Clearinghouses: Evaluation of Mental Health Interventions for Children
Evidence-based practice (EBP) requires a knowledge of the best current research as well as of practice wisdom and client preference. Internet registries and clearinghouses for outcome research on treatments are sources of evaluations of recent research and help identify high-quality research conclusions. This paper examines methods used by four registries and notes some problems of reporting. Ways to improve Internet registries are suggested.
There is increasing pressure for practitioners to use evidence-based practice (EBP), an approach that combines family preferences, practice wisdom and experience, and the current “best research” to guide treatment choices. Practitioners and clients together can work out the first two of these factors, but the third involves knowledge of outcome research that may require considerable time and access to professional journals. As a result, psychologists may turn to Internet sources of research information or to publications that summarize research work.
Internet sources range in quality from the highly variable Wikipedia, on which some articles are impeccably written and sourced but others are weak, to websites like www.wikia.psychology.com, which appear to encourage questionable statements in support of specific treatments. Treatments may also be described on proprietary websites that may provide excellent links to research support or may give no more than anecdotal evidence to support their claims. Systematic research syntheses (SRSs) also vary in quality, and journal articles assessing specific treatments do not readily allow readers to compare therapies with each other. Although the Cochrane Collaboration ( www.cochrane.org) has evaluated some psychological treatments, their reviews are much more often concerned with physical health.
As a result of these difficulties, it is common for practitioners to go to Internet “registries” or “clearinghouses” for which volunteers evaluate outcome research and assign scores indicating the amount and quality of research supporting specific treatments. The “registry” approach is depended upon to such an extent that a 2016 publication of the National Academies of Science, Engineering, and Medicine, Parenting matters, drew its recommendations about child psychotherapies from three registries. The publication made recommendations about Triple P, PCIT, and the Incredible Years, among others.
The present paper examines a series of “registries” or “clearinghouses”, including those used in the NAS Parenting matters report, considers the evaluative methods used by each, summarizes the levels of support reported for treatments, and notes where useful information about treatments has been omitted.
Two of the “registries” used in the NAS 2016 report were examined: the National Registry of Evidence Based Programs and Practices (NREPP; www.samhsa.gov.nrepp)and the California Evidence Based Clearinghouse for Child Welfare (CEBC: www.cebc4cw.org). The new Effective Child Therapy website (www.effectivechildtherapy.org ) was also considered, as was the British registry of the National Institute for Care and Health Excellence (NICE; https://nice.org.uk ).
For each registry, information on the website was used to explore a series of factors:
1. How mental health treatments for children were chosen to be listed on the website
2. Criteria used to evaluate information about outcome research
3. Methods of summarizing or ranking research quality
4. Numbers of treatments evaluated
5. Proportions of quality ranks assigned to the evaluated treatments
In addition, experiences of the present author in correspondence with two of the “registries” were noted.
NREPP, a service of the Substance Abuse and Mental Health Services Administration (SAMHSA), evaluates treatments whose proponents apply for assessment as well as those nominated by staff members. The following requirements must be met before a treatment will be evaluated.
1. Research or evaluation of the intervention has assessed mental health or substance use outcomes among individuals, communities, or populations OR other behavioral health-related outcomes on individuals, communities, or populations with or at risk of mental health issues or substance use problems.
of these outcomes has been demonstrated in at least one study using an experimental or quasi-experimental design.
Experimental designs require random assignment, a control or comparison
group, and pre- and post-intervention outcome assessments.
Quasi-experimental designs do not require random assignment, but do
require a comparison or control group and pre- and post-intervention
outcome assessments. Comparison/ control groups must be a no-treatment
control group, a wait-list control group, a treatment-as-usual comparison
group, or an intervention that is presumed to be ineffective or
substantially less effective than the intervention (e.g., a “placebo”
control or, in cases in which providing no treatment might be considered
unethical, less effective treatments, even if not treatment-as-usual, such
as “supportive therapy”). Studies with single-group, pretest-posttest
designs or single-group, longitudinal/multiple time series do not meet
this requirement, but will be considered to identify emerging programs and
practices for consideration in the Learning Center.
Comparative effectiveness trials, in which two interventions, both presumed to be equally effective, are compared, and studies in which the effects of the same intervention on various subpopulations are compared or in which various doses or components of the same intervention are compared will not be reviewed, but may be submitted as supporting documentation.
- The results of these studies have been published in a peer-reviewed journal or other professional publication, or documented in a comprehensive evaluation report, published within the previous 25 years. Comprehensive evaluation reports must include a review of the literature, theoretical framework, purpose, methodology, findings/results with statistical analysis and p values for significant outcomes, discussion, and conclusions (The NREPP Open Submission Process, 2016).
Changes in NREPP evaluative methods have been in the process of introduction since 2015. The website presently lists a large number of programs that were evaluated according to the old method (“legacy” programs), and a smaller number evaluated recently. The plan is to re-evaluate all listed programs by 2019.
The original NREPP evaluation looked at outcome research reports for the following factors:
1. Reliability of measures
2. Validity of measures
3. Intervention fidelity
4. Missing data and attrition
5. Confounding variables
6. Appropriateness of analysis
7. Adverse events
8. Plausibility of treatment, identification of therapeutic mechanism
Each treatment was given a numerical evaluation score based on these points.
Since 2015, the NREPP evaluation has examined these factors:
1. Rigor: design, intent-to-treat assignment, statistical precision, confounding of variables etc.
2. Effect size
3. Program fidelity
4. Conceptual framework
Treatments are then categorized as follows:
1. Effective: strong evidence of a favorable effect
2. Promising: sufficient evidence of a favorable effect
3. Ineffective: sufficient evidence of a negligible effect OR sufficient evidence of a possibly harmful effect
Note that the third category classes together treatments with little effect and those with some evidence of potential harm to children.
Of 387 NREPP-listed programs, 205 are primarily for children and adolescents. Ten of the 205 program reports mention possible adverse events as reported by the program developers. The mean rating for handling of confounded variables in programs listed on NREPP is 2.6 out of 4.0, with a range from 4.0 to 0.0, but this was reported in this form only for the “legacy” programs.
In spite of the apparently strict criteria for admission into the registry, the NREPP list includes Thought Field Therapy, a treatment long rejected as implausible and without evidentiary support (Pignotti, 2005).
The California Evidence-Based Clearinghouse for Child Welfare (CEBC; www.cebc4cw.org) is a searchable registry funded by the California Department of Social Services and the National Association of Public Child Welfare Administrators. CEBC evaluates programs that have strong empirical support, that are used often in California, or that are being heavily marketed in California. Evaluation is based on materials submitted by program developers.
Programs are evaluated and rated according to the following scale (for further details, see www.cebc4cw.org/files/OverviewOfTheCEBCScientificRatingScale.pdf).
1. Well-Supported by Research Evidence
2. Supported by Research Evidence
3. Promising Research Evidence
4. Evidence Fails to Demonstrate Effect
5. Concerning Practice
NR. Not able to be Rated on the CEBC Scientific Rating Scale
CEBC lists descriptions and findings about 286 programs, primarily for children and adolescents. Of listed programs, 26 were “non-responders” to inquiries about empirical support. Programs are rated according to stated criteria from 1 (excellent supportive evidence; 21 programs) to 5 (concerning; 0 programs, despite connections of some listed programs with adverse events), or Not Rated when supportive material is insufficient (77 programs). Seventy programs were rated “promising”.
CEBC was responsive to the present author’s query about a program, Circle of Security, that was listed as “well-supported” although published research reports did not agree with this evaluation. CEBC took down the existing statements about Circle of Security and a representative said that further material would be requested from the program developers.
The United Kingdom National Health Service’s National Institute for Health and Care Excellence (NICE; https://nice.org.uk) provides guidance relevant to a number of UK policies and practices. For the purposes of the present paper, its important feature is an evidence search service (https://www.evidence.nhs.uk). The NICE evidence search enables users to search for information on child psychotherapies, of which 2024 are listed (although a number of these focus on pharmaceutical treatment of childhood mental illness).
NICE warns users of the uncertainties inherent in the material provided: “In relation to the NICE Evidence Search only: It is in the nature of scientific debate that not all authors will agree on all matters. Further, published papers may be in error, or superseded by later research. Users should be aware that papers accessed through the website on the same subject-matter may report different results or conclusions. It is the user's sole responsibility to assess all evidence and to reach a decision informed by it. The website is designed only to assist access to some of the material relevant to decision-making. NICE Evidence Search only facilitates access to evidence published by third parties and no representation is made as to completeness, accuracy or fitness for purpose” (“Evidence Search”, 2016). In addition, the website notes that some sources of information have been accredited by NICE, but that this accreditation or guarantee does not extend to specific information provided. Although some items on the evidence search refer to the need for randomized controlled trials, no criteria for listing information about treatments are provided.
Sources of information on the NICE evidence search include the Social Care Institute for Excellence (SCIE; www.scie-socialcareonline.org.uk), a database of social work information that is said to be updated frequently by a team of information specialists. As well as journal article summaries, sources for the evidence search include the Database of Abstracts of Reviews of Effects (DARE).
Searching NICE for the treatment Dyadic Developmental Psychotherapy (DDP; Becker-Weidman & Hughes, 2008), a therapy that has been the subject of some argument about claims to be evidence-based, yields a 2006 summary by SCIE that repeats the claim that DDP is an evidence-based treatment. Correspondence with NICE by the present author did not produce any changes in this item. However, a search on SCIE itself shows the inclusion of a series of critiques of DDP as well as the 2006 summary. This suggests that NICE is not able to keep up-to-date on information about treatments, as indeed its warning message states.
A search of adverse effects of child psychotherapies yielded no references to risks of psychotherapy but gave examples of risks of medication.
Effective Child Therapy
Effective Child Therapy (www.effectivechildtherapy.org) is a recent registry project of the Society of Clinical Child and Adolescent Psychology (Div. 53 of the American Psychological Association). It is also supported by the Association for Behavioral and Cognitive Therapies. Effective Child Therapy lists programs as evidence-based according to rating methods used in evidence base update articles in the Journal of Clinical Child and Adolescent Psychology ( see Southam-Gerow & Prinstein, 2014.). These methods include criteria for well-established, probably effective, possibly effective, experimental, and ineffective treatments; none of these designations include consideration of adverse events. To understand the selection of programs to be evaluated and specific information used in evaluation, a reader would have to have access to the evidence base update articles that are the sources of the assessments. The site provides a listing of treatments for 19 disorders, with 82 uses of treatments evaluated (this number counts some treatments more than once, as the presentation shows the effectiveness of a treatment for a specific disorder). In 24 cases, a treatment is reported as having well-established effectiveness for a disorder; in 34 cases, a treatment is deemed probably effective; in 34 cases, a treatment is called possibly effective; in 40 cases, treatments are evaluated as experimental or with unknown support; and in 9 cases, treatments are said to be ineffective. Because the evaluative methods do not include assessments of adverse events, no treatment is described as harmful.
Internet registries or clearinghouses are useful but not perfect in their provision of the current “best research” that should be a factor in EBP. Because there are many people who want to use these sources for a variety of purposes, some improvements are needed.
NREPP set an example that could well be followed by other registries when it asked for examination of the plausibility or possible mechanism assumed by developers of a treatment. This factor, now described by NREPP as a conceptual framework, is not specifically addressed by other registries.
It is notable that registries tend to omit references to adverse events or to potential harm resulting from evaluated treatments, in some cases using the same category for weakly-supported positive effects and for potential for harm. This seems to be a failure to keep up with increasing concerns about the potential harmfulness of some child mental health treatments, which have grown along with a more general awareness of the possible iatrogenic effects of both physical treatments and psychotherapies. Lilienfeld (2007) considered both adult and child interventions as they might fit into the category of potentially harmful treatments (PHTs) .Lilienfeld (2007) suggested this term, PHT, for psychological interventions that were known to have caused or been associated with adverse events, or for treatments that might logically be expected to cause adverse events in some cases. Lilienfeld (2007) operationalized treatments as PHTs when they met three criteria: 1) demonstrated psychological or physical harm to clients or others, 2) enduring harmful effects, and 3) replicated evidence of harmful effects by independent research groups. Dimidjian and Hollon (2010) pursued these issues in a paper that discussed the concept of harm in psychotherapy and offered a distinction between treatments that are harmful and those that are simply unhelpful. Dimidjian and Hollon noted that a treatment may worsen outcomes both for the target problem and for other domains (including the creation of new problems) , that a treatment can have both helpful and harmful effects (again, with the possible creation of new problems), that an outcome can be considered helpful or harmful in different ways when seen from different perspectives, that outcomes may be initially harmful and later beneficial or the other way around, that outcomes of a treatment may be harmful for some patients but not all, that misuse of a beneficial treatment may cause harm, and that errors about benefits and risks may cause harm by preventing the use of a beneficial treatment. Although Dimidjian and Hollon did not comment on this, it may be added that in the case of mental health interventions for children, even an “unhelpful” treatment may also cause indirect harm by wasting family resources to the detriment of other family members as well as of the treated child. In spite of these developing concerns, however, Internet sites seem to be lessening attention to risks associated with treatments; the changing evaluative methods of NREPP are an example of this problem.
In addition to omitting attention to risks of treatments as well as to their benefits, Internet registries and clearinghouses generally fail to provide information about the cost-benefit analyses that may be important factors for choice of treatments in the present economy. Some of this information is available (Washington State Institute for Public Policy, 2016), together with assessments of the probability of benefit of a given treatment. For example, PCIT for disruptive children has been estimated to result in a benefits-minus-costs sum of $1808 and to have a probably of causing benefits of 79%; cognitive behavioral therapy models for child trauma have been estimated to have a benefit-minus-cost sum of $6550 and to have a probability of benefit of 100%; some other treatments are estimated to have costs that exceed their benefits and to have a probability of benefits as low as 4%. For registries to include this information where possible would be helpful to users, especially those working on projects like the Parenting matters book.
Internet registries tend to depend on volunteer evaluators and/or to base evaluations on material provided by program developers. They may or may not be responsive to critiques and queries from outsiders. In some cases, such as NICE, they draw material and conclusions from other organizations which take responsibility for assessing evidence and for updating recommendations when new evidence becomes available. Although no registry can be expected to do a perfect job or to keep up with new evidence in real time, it could be useful for registries to use volunteers primarily to search for emerging evidence and to employ trained staff to do the job of evaluating outcome research on child mental health interventions.
Becker-Weidman, A., & Hughes, D. (2008). Dyadic Developmental Psychotherapy: An evidence-based treatment for children with complex trauma and disorders of attachment. Child and Family Social Work, 13(3), 329-337.Dimidjian, S., & Hollon, S. (2010). How would we know if psychotherapy were harmful?American Psychologist, 65, 21-33.
“Evidence Search” (2016). Retrieved Sept. 3, 2016 from https://www.evidence.nhs.uk
Lilienfeld, S.O. (2007). Psychological treatments that cause harm. Perspectives on Psychological Science, 2(1), 53-70.
National Academies of Science, Engineering, and Medicine. (2016). Parenting matters: Supporting parents of children aged 0-8. Washington, DC: The National Academies Press. Retrieved Sept. 2, 2016 from http://www.nap.edu/read/218681/
Pignotti, M. (2005). Thought Field Therapy Voice technology versus random meridian point sequences: A single blind controlled experiment. Scientific Review of Mental Health Practice, 4, 38-47.
Southam-Gerow, M.A., & Prinstein, M.J. (2014). Evidence base updates: The evolution of the evaluation of psychological treatments for children and adolescents. Journal of Clinical Child and Adolescent Psychology, 43(1), 1-6.
“The NREPP Open Submission Process”. (2016). Retrieved Sept. 5, 2016 from www.nrepp.samhsa.gov/04b_reviews_open.aspx .
Washington State Institute for Public Policy (2016). Benefit-cost results. Retrieved Sept. 3, 2016 from www.wsipp.wa.gov/BenefitCost .