Randomization? Yes; Isolated Variable? No: Claims About QST for Autism

Last week, I received a message forwarded from an old colleague, with a link to an article in an open access journal. Here’s the reference: Silva, L.M.T., Schalock, M., Gabrielsen, K.R., Budden, S.S., Buenrostro, M., & Hortin, G. (2015). Early intervention with a parent-delivered massage protocol directed at tactile abnormalities decreases severity of autism and improves child-to-parent interactions: A replication study. Autism Research and Treatment. http://dx.doi.org.10.1155/2015/904585.

I was curious about this because I know my old colleague is a fan of DIR/Floortime, and DIR/Floortime has been accompanied quite frequently by various alternative, non-evidence-based treatments like SIT. The Silva et al study was introduced as dealing with a treatment said by the authors to be shown to be evidence-based--  QST, or Qigong Sensory Treatment. QST involves a whole-body massage usually done at bedtime. According to the Silva et al article, “the protocol has 12 parts that follow the acupuncture channels down the front and back of the body. Massage is carried out in a downward direction towards the hands and feet in the direction of capillary blood flow.” Children who were treated with QST experienced a massage lasting about 15 minutes on most nights for a five month period. They also received 20 sessions of massage by therapist.

Children were recruited to the study by advertising and by invitation letters to those who were already receiving autism services from the state of Oregon. After screening, children were assigned at random either to a treatment group, receiving the treatment just described, or to a waitlist group, receiving no treatment other than what they had been getting before (two children were receiving ABA treatment). . Evaluation scores for the two groups were compared to each other statistically at the end of a five month treatment period. Professional evaluators did not know which group a child was in, but information was also collected from parents, who, of course, were aware of the child’s treatment status.

The study authors concluded that QST could be considered an evidence-based treatment, because significant advantages for the treatment group were found in this randomized, controlled study. But here is where it all goes agley, because although randomization is necessary for collection of evidence supporting a therapy, it is not sufficient.

There is a purpose to randomization. It is not simply a ritual carried out so the research gods will smile upon a study. The reason for randomization is that it is a step that helps to isolate a variable—to make sure, in the old phrase, that we are comparing apples to apples, not to oranges or tutti-frutti. Randomizing participants to groups is useful only if the researchers have previously designed the groups so that they resemble each other closely on as many particulars as can be managed, except for the specific factor whose effects are being studied. If the groups are given very different experiences, it is impossible to nail down which factor  or factors caused the observed outcome.

Randomizing the participants in this study made sure that there were not more boys than girls, or more older than younger children, or more or less severely-affected children, in the treatment group than in the comparison group. That was an excellent step and made sure that  the treatment was not made to look good just because, for example, the children in the treatment group were less troubled to start with. But then the problem began, because the children in the treatment group did not have experiences that were different in a simple way from the experiences of the control group. Yes, one group got QST, and the other didn’t, but when we unpack those facts, we find that there were a number of unnecessary differences between the groups’ experiences—differences that confuse the issue of the effects of the treatment.

The group of children that got QST, with its downward massage along acupuncture channels, also received more than 30 hours of intimate contact with their parents, and 20 hours of contact with massage therapists, over about 150 days. Parents were taught to modulate their massage efforts in response to a child’s apparent acceptance or rejection of a kind of touch, so the children also experienced some number of events of successful communication with a parent--  possibly an important factor, considering the difficulty of autistic children with communication.

Parents of children in the treatment group received warm treatment and training from the therapists who taught them to do massage, and frequent repeated contacts as the researchers tried to keep the parent massage at the desired standard. For parents suffering from the stresses of caring for autistic children, the social and emotional support of the research group, and the sense of hope engendered by learning this new approach, may have helped them tolerate both ordinary daily problems and the constant concern about their children’s future; increased hope and tolerance could improve the parent’s ability to interact positively with a difficult child. Like the children, the parents could well have been affected by nightly interactions involving a predictable ritual and providing a chance to experience success. All of these factors had the potential for actually helping the parent do a better job with the child in general (separately from the QST) and to regard the child more positively, perhaps yielding better parent evaluations.

What about the waitlisted control group? The children may not have been very much aware that a new form of treatment might be available, so they may have been affected only by the absence of QST and of the related factors described above.  But what about the parents? Presumably they were eager to try QST, or they would not have gone to the trouble of answering the research invitations. When assigned to the waitlist, some may have reacted philosophically, realizing that their children would get a later chance for treatment.  However, many may have been deeply disappointed and contemplated with distress another five months in which their children--  whom nothing had helped much so far—were still not getting the help that had seemed to be on the horizon. Discouraged parents with waitlisted children may have said to each other, “I was so excited about this new thing, but now we have to wait some more. I don’t have much hope that our kids will ever get any better.” Such reactions could affect both attitudes and behavior toward children, and evaluations of children’s conditions.

Naturally, I can’t say which, if any, of these things actually did happen. But I can say how the researchers could have designed the study so that the results had much more meaning. While the treatment group received QST, the control group could have been given a similar massage treatment that did not involve the “channels” claimed as the essence of the treatment.  If parents were simply taught one or the other massage, and were not informed which was the “real” QST and which was a sham, the problem of blinding and of parent attitudes would be solved. A study in which QST outcomes were significantly better than sham outcomes would justify the researchers’ claim that QST should be considered evidence-based.

Unfortunately,  Silva et al are not alone in erroneously claiming that randomization and control alone are sufficient to provide evidence of a treatment’s effectiveness. There is a lot of this around just now.  Studies of DIR/Floortime have shown very similar problems. A few months ago I pointed out on this blog that the Bucharest Early Intervention Study failed to isolate the variable under investigation, even though randomization was used.

Catchphrases like randomization, control, RCT, evidence-based treatment, are deceptive for those who do not fully understand them. Like the alcohol rub for fever discussed long ago by Dr. Spock, they smell important, but they can be used incorrectly. The difficulty is that somebody really has to read the work in order to know whether, like alcohol, it evaporates on contact.