When Conclusions Outrun Evidence: Karyn Purvis's Camp Study

Having had many arguments with people who promote unsupported theories, I have to state approval for those who go through the tedious work of collecting empirical evidence that can support or disconfirm a hypothesis. However, those who have collected evidence should not think they are finished with the job. They must still analyze the evidence and draw logical conclusions--  and, if in the course of this work it appears that they did not design the study well to begin with, they need to start over. Otherwise, conclusions and support for theory will go beyond what the evidence shows, and readers may be deceived into accepting conclusions when they should not do so.

A case in point is a recent publication by Karyn Purvis, L.Brooks McKenzie, David Cross, and Erin Becker Razuri, in the [theoretically] peer-reviewed Journal of Child and Adolescent Psychiatric Nursing (“A spontaneous emergence of attachment behavior in at-risk children and a correlation with sensory deficits”, 2013, 26(3), pp. 165-172). The flaws in this article raise many questions about the efficacy of peer review--  although I suppose that if a reviewer is really at a peer level with a confused author, we can hardly expect the reviewer to notice problems.

The Purvis et al paper discusses a therapeutic day camp experience for a group of children recruited from local support groups and therapists (in other words, a group whose parents believed this camp was a good idea and likely to have positive effects on the children). Data were collected from 18 children whose ages ranged from 3 to 14 years (in other words, a small group with such a wide age range that it is difficult to know either how they could all have been given the same treatment, or how different treatments would have been chosen). All had been adopted and “many” (but how many, or which, remains unclear) were institutionalized as infants or toddlers. These children and adolescents were tested on a range of tests--  including the much-rejected Randolph Attachment Disorder Questionnaire and the questionable Beech Brook Attachment Disorder Checklist--  that looked at social and emotional capacities and at sensory abilities as noted by parents and by occupational therapists.

Testing was done before and after the camp experience. In other words, the design of this study was at the low level of evidence characteristic of the pre- and post-treatment design. This design admits multiple confounding variables into the situation, so that upon completion it is impossible to tell whether a treatment or some other factor(s) caused an outcome.  In this case, Purvis and her colleagues concluded that the treatment itself caused various improvements in the children’s behavior and abilities. They did not discuss the possibility that parents who believe that their children are receiving a helpful treatment may themselves change both their responses to and their evaluations of the children. They ignored the fact that parents as well as children in this study might have been directly affected, as each camp day began with direct questions to the parents in the children’s presence: “May I be the boss of your child while you are gone?” and “If your child asks for a hug today, may I give them a hug?” These questions communicated to both parent and child beliefs about the importance of the parent in the child’s life and positive attitudes about the relationship between the child and the staff member. Purvis et al also neglected to consider that 6 weeks of camp experience, including interactions with a college student assigned to be the child’s buddy, could make a difference to children’s experience in the camp situation and therefore to the impact of that experience on behavior at home. These confounding variables could well be responsible for the reported emergence of attachment behaviors; it’s notable, by the way, that there was no measure of reduced attachment behaviors, which might well be expected and appropriate for the older children.

There is much more to be said about the design and implementation of the study, particularly about the statistical analysis. However, I would like to move on to comment on certain assumptions made by Purvis and her colleagues and often shared by occupational therapists and some educators. These have to do with the function of the vestibular system and the effects of stimulating the system. As it happens, I did my Ph.D. thesis on the effects of vestibular stimulation on other sensory processes, and would be the first to concede that when there is very little sensory information to use, strong and directed vestibular stimulation influences perceptual judgments. However, outside the laboratory, there are few situations where there is a minimum of information (e.g., being in the dark) and directed vestibular stimulation like accelerated rotation in one direction. Most experience of vestibular stimulation is highly variable and undirected, and it happens literally all the time. The vestibular system is always affected by the pull of gravity unless a person is in outer space, and it is simultaneously affected by every movement of head and body or head alone, self-produced or caused by the environment. Infants and young  children like and are interested and soothed by forms of vestibular stimulation, just as they are by rhythmic sound, touch, or changing visual stimulation.

It is certainly true, as Purvis says, that “each time a child is picked up, the fluid in the … semicircular canal shifts, creating what has become known as vestibular input” (p. 170). But this also happens every time the child turns the head or the whole body, turns over in sleep, walks, jumps up and down, crawls, changes sitting position, etc., etc.  There are three semicircular canals on each side of the head, each specialized to respond to a particular direction of rotation, and there are also gravitational receptors that send messages along the auditory-vestibular nerve even if there is no rotational head movement. It is inconceivable that a living person existing in a gravitational field could be without vestibular activity. There is no evidence that extra vestibular activity causes better development, or indeed that therapeutic additions give more than a slight increment to children’s self-produced vestibular activity. These are beliefs that have been promoted without evidence by people like Ayers and Wilbarger, whom Purvis cites. (Incidentally, isn’t it a bit inconsistent that we want these children to be less active, when their activity gives them vestibular stimulation? Or is it that only the kind therapists give is any good?)

As the Reuters study a few weeks ago, showed us, adoptive families sometimes need extra help. But it seems to me a shame that a journal has published an article that states unsupported conclusions about the kinds of help that may actually be effective. Readers, please do take care about accepting some of these ideas! There are evidence-based treatments that can be of real help.