Wednesday, April 18, 2018
How Do We Know Whether an Intervention Is Effective? (E.g., PA Treatments)
There’s a lot of discussion right now about whether psychological interventions are or are not evidence-based (or empirically-supported) treatments. Even when there is strong evidence that a treatment is efficacious—causes good outcomes under ideal circumstances—there are further demands for evidence that it is effective and causes good outcomes in the less-than-ideal settings of the real world.
The strongest evidence in either case depends on research that has used randomized controlled trials. RCTs (not to be confused with residential treatment centers!) are studies in which people who have received the treatment being tested are compared to a control group of people who have not received it (and may have had no treatment, or had a well-established different treatment, or have been placed on a wait-list for the treatment being tested). Because this kind of research is randomized, prospective patients do not get to choose which group they want to be in, but are assigned to one or the other in some objective way like the flip of a coin or the use of a list of random numbers. The point here is to eliminate the bias that might be present if people got to choose the treatment they want—just believing that something will be good for them, or expecting that a treatment will be effective, can be enough to change people’s moods and behavior. (If we’re talking about treatments for children and adolescents, parents may be affected by their expectations about a treatment so that they behave differently toward their children if they think they are getting that treatment, or see their children’s behavior differently if they think the children are getting a good treatment, or, of course, both of these things.)
Having a control or comparison group as part of the research design adds to what is called the internal validity of the study. Internal validity is also influenced by factors like having a blinded evaluation, or assessment of children by people who do not know them or know what treatment they have been receiving or will receive. When parents, who know both of these things to some extent (even if not told their children’s treatment, they may figure it out from things the child tells them), are involved in evaluation of the children’s response to treatment, other biases can creep in; parents are the ones to say whether they liked a treatment, but can not say objectively whether it helped their children. A number of alternative therapies, like Love & Logic training for parents, make the mistake of asking parents whether their children have improved and then acting as if it is known what changes have really occurred in the children.
Many studies that claim to give evidence supporting unconventional, alternative therapies are more than one step below RCTs. They look very simply at measures (often parent opinions) taken before and after treatment. Does this seem like a good idea? Well, only if you think that children and adolescents do not change unless they are made to change by an intervention!
I recently had pointed out to me the statement by a guardian ad litem that changes she heard about in a girl between ages 11 and 13 could only have been caused by a PA treatment method the girl had experienced. It would appear that the GAL thought that nothing happened in those two years other than the treatment and separation from the preferred parent. But in fact much had happened.
A small amount of the change reported was no doubt due to adjustment over a couple of post-divorce years, during which one parent had remarried. A much larger factor, causing changes of all kinds, physical and mental, was the occurrence of puberty for this girl—just under 13years being the average age of menarche (first menstruation) in the Western world today. Puberty in both sexes is associated with dramatic social and cognitive changes and the redirection of “attachment” and positive social attitudes toward peers and away from parents. Height and weight increase, breasts develop, hair and skin change, all producing a mixture of delighted and anxious feelings about the self, and further reducing (though not eliminating) the adolescent’s concern about relationships with parents.
Maturation, the series of developmental changes that is largely governed by genetic programs, is rapid and intense during this period when “nothing happened” except the PA intervention. Without good research design that takes maturation into account, studies of PA or other interventions for children and adolescents are wrong if they claim they show the influence of their intervention. Because children and adolescents are always changing—most rapidly around puberty, beginning about age 9 in girls—every study of a psychological treatment for them needs to have a control group, similar to the treated group but nor receiving the treatment, whose characteristics will show us the natural maturational changes that occur during an age period. The effect of the PA or other intervention is shown (roughly speaking) by subtracting:
Changes following intervention (maturation + intervention) - Changes with maturation alone =
Changes attributable to intervention
There would be many other technical details to be considered in designing a good study of PA or other interventions, but the two mentioned here, blind evaluation and a control for maturational change, are absolutely critical.