I wrote the following paper for a conference about a year ago, and recently a colleague asked whether it was on line anywhere-- so here it is, for those interested. (Since writing this, I have come across cases of confusion about the meaning of the CEBC evaluations and the conflation of "relevance" scores with research scores.)
Internet Registries and Clearinghouses: Evaluation of
Mental Health Interventions for Children
Jean Mercer
Stockton University
Abstract
Evidence-based practice (EBP) requires a knowledge of
the best current research as well as of practice wisdom and client preference.
Internet registries and clearinghouses for outcome research on treatments are
sources of evaluations of recent research and help identify high-quality
research conclusions. This paper examines methods used by four registries and
notes some problems of reporting. Ways to improve Internet registries are
suggested.
There is increasing
pressure for practitioners to use evidence-based practice (EBP), an approach
that combines family preferences, practice wisdom and experience, and the
current “best research” to guide treatment choices. Practitioners and clients together can work
out the first two of these factors, but the third involves knowledge of outcome
research that may require considerable time and access to professional
journals. As a result, psychologists may turn to Internet sources of research
information or to publications that summarize research work.
Internet sources range in
quality from the highly variable Wikipedia, on which some articles are
impeccably written and sourced but others are weak, to websites like www.wikia.psychology.com,
which appear to encourage questionable statements in support of specific
treatments. Treatments may also be described on proprietary websites that may
provide excellent links to research support or may give no more than anecdotal
evidence to support their claims. Systematic research syntheses (SRSs) also
vary in quality, and journal articles assessing specific treatments do not
readily allow readers to compare therapies with each other. Although the Cochrane Collaboration ( www.cochrane.org)
has evaluated some psychological treatments, their reviews are much more often
concerned with physical health.
As a result of these
difficulties, it is common for practitioners to go to Internet “registries” or
“clearinghouses” for which volunteers evaluate outcome research and assign
scores indicating the amount and quality of research supporting specific
treatments. The “registry” approach is depended upon to such an extent that a
2016 publication of the National Academies of Science, Engineering, and
Medicine, Parenting matters, drew its
recommendations about child psychotherapies from three registries. The
publication made recommendations about Triple P, PCIT, and the Incredible
Years, among others.
The present paper
examines a series of “registries” or “clearinghouses”, including those used in
the NAS Parenting matters report,
considers the evaluative methods used by each, summarizes the levels of support
reported for treatments, and notes where useful information about treatments
has been omitted.
Method
Two of the “registries”
used in the NAS 2016 report were examined: the National Registry of Evidence
Based Programs and Practices (NREPP; www.samhsa.gov.nrepp)and
the California Evidence Based Clearinghouse for Child Welfare (CEBC: www.cebc4cw.org).
The new Effective Child Therapy website (www.effectivechildtherapy.org
) was also considered, as was the British registry of the National Institute
for Care and Health Excellence (NICE; https://nice.org.uk
).
.
For each registry,
information on the website was used to explore a series of factors:
1. How mental health treatments for children were
chosen to be listed on the website
2. Criteria
used to evaluate information about outcome research
3. Methods
of summarizing or ranking research quality
4. Numbers
of treatments evaluated
5. Proportions
of quality ranks assigned to the evaluated treatments
In addition, experiences
of the present author in correspondence with two of the “registries” were
noted.
Results
NREPP
NREPP, a service of the
Substance Abuse and Mental Health Services Administration (SAMHSA), evaluates
treatments whose proponents apply for assessment as well as those nominated by
staff members. The following requirements must be met before a treatment will
be evaluated.
1.
Research
or evaluation of the intervention has assessed mental health or substance use outcomes among individuals,
communities, or populations OR other behavioral health-related outcomes on
individuals, communities, or populations with or at risk of mental health
issues or substance use problems.
- Evidence
of these outcomes has been demonstrated in at least one study using an experimental or quasi-experimental design.
Experimental designs require random assignment, a control or comparison
group, and pre- and post-intervention outcome assessments.
Quasi-experimental designs do not require random assignment, but do
require a comparison or control group and pre- and post-intervention
outcome assessments. Comparison/ control groups must be a no-treatment
control group, a wait-list control group, a treatment-as-usual comparison
group, or an intervention that is presumed to be ineffective or
substantially less effective than the intervention (e.g., a “placebo”
control or, in cases in which providing no treatment might be considered
unethical, less effective treatments, even if not treatment-as-usual, such
as “supportive therapy”). Studies with single-group, pretest-posttest
designs or single-group, longitudinal/multiple time series do not meet
this requirement, but will be considered to identify emerging programs and
practices for consideration in the Learning Center.
Comparative effectiveness trials, in which two interventions, both presumed to be equally effective, are compared, and studies in which the effects of the same intervention on various subpopulations are compared or in which various doses or components of the same intervention are compared will not be reviewed, but may be submitted as supporting documentation. - The
results of these studies have been published in a peer-reviewed journal or other professional publication, or
documented in a comprehensive evaluation report, published within the
previous 25 years.
Comprehensive evaluation reports must include a review of the literature,
theoretical framework, purpose, methodology, findings/results with
statistical analysis and p values for significant outcomes, discussion,
and conclusions (The NREPP Open Submission Process, 2016).
Changes in NREPP evaluative
methods have been in the process of introduction since 2015. The website
presently lists a large number of
programs that were evaluated according to the old method (“legacy” programs),
and a smaller number evaluated recently. The plan is to re-evaluate all listed
programs by 2019.
|
The original NREPP evaluation looked at outcome
research reports for the following factors:
|
||
|
|
|
|
1.
Reliability of measures
2.
Validity
of measures
3.
Intervention fidelity
4.
Missing data and attrition
5.
Confounding variables
6.
Appropriateness of analysis
7.
Adverse events
8.
Plausibility of treatment, identification
of therapeutic mechanism
Each
treatment was given a numerical evaluation score based on these points.
Since 2015, the NREPP evaluation has examined these
factors:
1. Rigor:
design, intent-to-treat assignment, statistical precision, confounding of
variables etc.
2. Effect
size
3. Program
fidelity
4. Conceptual framework
Treatments are then categorized as follows:
1. Effective:
strong evidence of a favorable effect
2. Promising:
sufficient evidence of a favorable effect
3. Ineffective:
sufficient evidence of a negligible effect OR sufficient evidence of a possibly
harmful effect
4. Inconclusive
Note that the third category classes together
treatments with little effect and those with some evidence of potential harm to
children.
Of 387 NREPP-listed programs,
205 are primarily for children and adolescents. Ten of the 205 program reports
mention possible adverse events as reported by the program developers. The mean
rating for handling of confounded variables in programs listed on NREPP is 2.6 out of 4.0, with a range from 4.0 to
0.0, but this was reported in this form only for the “legacy” programs.
In spite of the apparently strict criteria for admission into the
registry, the NREPP list includes Thought Field Therapy, a treatment long
rejected as implausible and without evidentiary support (Pignotti, 2005).
CEBC
The California
Evidence-Based Clearinghouse for Child Welfare (CEBC; www.cebc4cw.org)
is a searchable registry funded by the California Department of Social Services
and the National Association of Public Child Welfare Administrators. CEBC
evaluates programs that have strong empirical support, that are used often in
California, or that are being heavily marketed in California. Evaluation is
based on materials submitted by program developers.
Programs are
evaluated and rated according to the following scale (for further details, see www.cebc4cw.org/files/OverviewOfTheCEBCScientificRatingScale.pdf).
1. Well-Supported
by Research Evidence
2. Supported by Research
Evidence
3.
Promising Research Evidence
4. Evidence Fails to
Demonstrate Effect
5. Concerning Practice
NR. Not able to be Rated
on the CEBC Scientific Rating Scale
CEBC lists
descriptions and findings about 286 programs, primarily for children and
adolescents. Of listed programs, 26 were “non-responders” to inquiries about
empirical support. Programs are rated according to stated criteria from 1
(excellent supportive evidence; 21 programs) to 5 (concerning; 0 programs,
despite connections of some listed programs with adverse events), or Not Rated
when supportive material is insufficient (77 programs). Seventy programs were
rated “promising”.
CEBC was responsive to the present author’s query
about a program, Circle of Security, that
was listed as “well-supported” although published research reports did not
agree with this evaluation. CEBC took down the existing statements about Circle
of Security and a representative said that further material would be requested
from the program developers.
NICE
The United Kingdom
National Health Service’s National Institute for Health and Care Excellence
(NICE; https://nice.org.uk)
provides guidance relevant to a number of UK policies and practices. For the purposes
of the present paper, its important feature is an evidence search service (https://www.evidence.nhs.uk).
The NICE evidence search enables users to search for information on child
psychotherapies, of which 2024 are listed (although a number of these focus on
pharmaceutical treatment of childhood mental illness).
NICE warns users of the
uncertainties inherent in the material provided: “In
relation to the NICE Evidence Search only: It is in the
nature of scientific debate that not all authors will agree on all matters.
Further, published papers may be in error, or superseded by later research.
Users should be aware that papers accessed through the website on the same
subject-matter may report different results or conclusions. It is the user's
sole responsibility to assess all evidence and to reach a decision informed by
it. The website is designed only to assist access to some of the material
relevant to decision-making. NICE Evidence Search only facilitates access to
evidence published by third parties and no representation is made as to
completeness, accuracy or fitness for purpose” (“Evidence Search”, 2016). In
addition, the website notes that some sources of information have been
accredited by NICE, but that this accreditation or guarantee does not extend to
specific information provided. Although some items on the evidence search refer
to the need for randomized controlled trials, no criteria for listing
information about treatments are provided.
Sources of
information on the NICE evidence search include the Social Care Institute for
Excellence (SCIE; www.scie-socialcareonline.org.uk), a database of social work information that is said to be
updated frequently by a team of information specialists. As well as journal
article summaries, sources for the evidence search include the Database of
Abstracts of Reviews of Effects (DARE).
Searching
NICE for the treatment Dyadic Developmental Psychotherapy (DDP; Becker-Weidman
& Hughes, 2008), a therapy that has been the subject of some argument about
claims to be evidence-based, yields a 2006 summary by SCIE that repeats the
claim that DDP is an evidence-based treatment. Correspondence with NICE by the
present author did not produce any changes in this item. However, a search on
SCIE itself shows the inclusion of a series of critiques of DDP as well as the
2006 summary. This suggests that NICE is not able to keep up-to-date on
information about treatments, as indeed its warning message states.
A search of
adverse effects of child psychotherapies yielded no references to risks of
psychotherapy but gave examples of risks of medication.
Effective Child Therapy
Effective Child
Therapy (www.effectivechildtherapy.org) is
a recent registry project of the Society of Clinical Child and Adolescent
Psychology (Div. 53 of the American Psychological Association). It is also
supported by the Association for Behavioral and Cognitive Therapies. Effective
Child Therapy lists programs as evidence-based according to rating methods used
in evidence base update articles in the Journal
of Clinical Child and Adolescent Psychology ( see Southam-Gerow & Prinstein,
2014.). These methods include criteria for well-established, probably
effective, possibly effective, experimental, and ineffective treatments; none
of these designations include consideration of adverse events. To understand the
selection of programs to be evaluated and specific information used in
evaluation, a reader would have to have access to the evidence base update
articles that are the sources of the assessments. The site provides a listing
of treatments for 19 disorders, with 82 uses of treatments evaluated (this
number counts some treatments more than once, as the presentation shows the
effectiveness of a treatment for a specific disorder). In 24 cases, a treatment
is reported as having well-established effectiveness for a disorder; in 34 cases,
a treatment is deemed probably effective; in 34 cases, a treatment is called
possibly effective; in 40 cases, treatments are evaluated as experimental or
with unknown support; and in 9 cases, treatments are said to be ineffective.
Because the evaluative methods do not include assessments of adverse events, no
treatment is described as harmful.
Discussion
Internet registries
or clearinghouses are useful but not perfect in their provision of the current
“best research” that should be a factor in EBP. Because there are many people
who want to use these sources for a variety of purposes, some improvements are needed.
NREPP set an example
that could well be followed by other registries when it asked for examination
of the plausibility or possible mechanism assumed by developers of a treatment.
This factor, now described by NREPP as a conceptual framework, is not
specifically addressed by other registries.
It is notable that
registries tend to omit references to adverse events or to potential harm
resulting from evaluated treatments, in some cases using the same category for
weakly-supported positive effects and for potential for harm. This seems to be
a failure to keep up with increasing concerns about the potential harmfulness
of some child mental health treatments, which have grown along with a more
general awareness of the possible iatrogenic effects of both physical
treatments and psychotherapies. Lilienfeld (2007) considered both adult and
child interventions as they might fit into the category of potentially harmful
treatments (PHTs) .Lilienfeld (2007) suggested this term, PHT, for psychological interventions that were
known to have caused or been associated with adverse events, or for treatments
that might logically be expected to cause adverse events in some cases.
Lilienfeld (2007) operationalized treatments as PHTs when they met three
criteria: 1) demonstrated psychological or physical harm to clients or others,
2) enduring harmful effects, and 3) replicated evidence of harmful effects by
independent research groups. Dimidjian and Hollon (2010) pursued these issues
in a paper that discussed the concept of harm in psychotherapy and offered a
distinction between treatments that are harmful and those that are simply
unhelpful. Dimidjian and Hollon noted
that a treatment may worsen outcomes both for the target problem and for other
domains (including the creation of new problems) , that a treatment can have
both helpful and harmful effects (again, with the possible creation of new
problems), that an outcome can be considered helpful or harmful in different
ways when seen from different perspectives, that outcomes may be initially
harmful and later beneficial or the other way around, that outcomes of a
treatment may be harmful for some patients but not all, that misuse of a
beneficial treatment may cause harm, and that errors about benefits and risks
may cause harm by preventing the use of a beneficial treatment. Although
Dimidjian and Hollon did not comment on this, it may be added that in the case
of mental health interventions for children, even an “unhelpful” treatment may
also cause indirect harm by wasting family resources to the detriment of other
family members as well as of the treated child.
In spite of these developing concerns, however, Internet sites seem to
be lessening attention to risks associated with treatments; the changing
evaluative methods of NREPP are an example of this problem.
In addition to
omitting attention to risks of treatments as well as to their benefits,
Internet registries and clearinghouses generally fail to provide information
about the cost-benefit analyses that may be important factors for choice of
treatments in the present economy. Some of this information is available
(Washington State Institute for Public Policy, 2016), together with assessments
of the probability of benefit of a given treatment. For example, PCIT for
disruptive children has been estimated to result in a benefits-minus-costs sum
of $1808 and to have a probably of causing benefits of 79%; cognitive
behavioral therapy models for child trauma have been estimated to have a
benefit-minus-cost sum of $6550 and to have a probability of benefit of 100%;
some other treatments are estimated to have costs that exceed their benefits
and to have a probability of benefits as low as 4%. For registries to include this information
where possible would be helpful to users, especially those working on projects
like the Parenting matters book.
Internet registries
tend to depend on volunteer evaluators and/or to base evaluations on material
provided by program developers. They may or may not be responsive to critiques
and queries from outsiders. In some cases, such as NICE, they draw material and
conclusions from other organizations which take responsibility for assessing
evidence and for updating recommendations when new evidence becomes available. Although no registry can be expected to do a
perfect job or to keep up with new evidence in real time, it could be useful
for registries to use volunteers primarily to search for emerging evidence and
to employ trained staff to do the job of evaluating outcome research on child
mental health interventions.
References
Becker-Weidman, A., & Hughes, D.
(2008). Dyadic Developmental Psychotherapy: An evidence-based treatment for children with
complex trauma and disorders of attachment.
Child and Family Social Work, 13(3), 329-337.
Dimidjian,
S., & Hollon, S. (2010). How would we know if psychotherapy were harmful?American
Psychologist, 65, 21-33.
Lilienfeld, S.O. (2007). Psychological
treatments that cause harm. Perspectives
on Psychological Science, 2(1),
53-70.
National Academies of Science,
Engineering, and Medicine. (2016). Parenting
matters: Supporting parents of children aged 0-8. Washington, DC: The
National Academies Press. Retrieved Sept. 2, 2016 from http://www.nap.edu/read/218681/
Pignotti, M. (2005). Thought Field Therapy
Voice technology versus random meridian point sequences: A single blind
controlled experiment. Scientific Review
of Mental Health Practice, 4, 38-47.
Southam-Gerow, M.A., & Prinstein,
M.J. (2014). Evidence base updates: The evolution of the evaluation of
psychological treatments for children and adolescents. Journal of Clinical Child and Adolescent Psychology, 43(1), 1-6.
“The NREPP Open Submission Process”.
(2016). Retrieved Sept. 5, 2016 from www.nrepp.samhsa.gov/04b_reviews_open.aspx .
Washington State
Institute for Public Policy (2016). Benefit-cost results. Retrieved Sept. 3,
2016 from www.wsipp.wa.gov/BenefitCost .
