Why Can't They Tell Which Patient Will Kill Somebody?

Recently, someone commenting on a post on this blog asked me to explain why I had said that diagnosis of mental illness in research did not have to be, or at least was not, as accurate as was needed in treatment of an individual. The writer pointed out that therapists may make a diagnosis not so much because it is accurate but because it allows particular treatments or services to begin. This is certainly true, and it’s also true that psychosocial treatments are often directed at specific symptoms that are troubling rather than at some underlying condition that has been diagnosed. That’s a good idea in many cases, especially when the proposed condition, like Reactive Attachment Disorder, does not necessarily have the causes that are posited for it.

My remarks were not about how things actually are, or about the best they can be in light of various social and political pressures. Instead, I was thinking about the kinds of questions researchers and therapists are asking, and the ways these questions differ from each other. Researchers are almost invariably asking whether one group of people is different from another, or about what will happen to a group given one treatment, as compared with a group given another treatment. They expect some variability within groups and would be surprised and even suspicious if everyone in a group acted the same way. They also accept the fact that diagnostic measures vary in their accuracy.

Therapists, on the other hand, want to know what will be the effect on a particular person of a treatment or experience. They have much less wiggle room than researchers do, especially in cases where a patient may or may not behave violently. We are all aghast every time we read that a formerly violent patient was allowed a weekend pass from a hospital, went home, and chopped up his mother with an ax. “Why didn’t they know that would happen?," we demand.

A useful article in Science (“What is the question?, by Jeffrey Leek and Roger Peng, 20 March 2015, pp. 1314-1315) provides some ways of thinking about these issues. Leek and Peng even give a great flowchart, and I am going to shamelessly follow their description of making decisions about the kinds of questions that are being asked by researchers, therapists, and lots of other people.

The first question Leek and Peng ask about how people think about information they have available is, “did you summarize the data?” If this hasn’t been done--  for example, if there are only anecdotes or testimonials in use—there is no data analysis, and no prediction can be done. (For example, about whether a particular treatment produced a better outcome than another did. )

If the information was summarized, but reported without any interpretation, this was a descriptive approach, but again no prediction can be done.

What if the information was not only summarized, but interpreted—but there was no attempt to decide whether the patterns seen would be repeated in other circumstances? Work of this kind is exploratory. Whatever patterns or connections exist between factors (like treatments experienced), they still need to be confirmed by more work.

Did the study quantify the differences observed and calculate the probability that they would be repeated? If so, there are further questions to be asked. The first one is whether someone is trying to figure out how the average of one measurement affects another measurement. If this is not being done, the next question is whether the goal is to predict measurements for individuals. No? Then the study is an inferential one, which just looks for relationships between factors. Yes? The study is a predictive one. It attempts to predict what will happen with a single individual--  but without being able to understand how or why effects occur, and therefore without real certainty.

Suppose there is an effort to find out how changing the average of one measurement changes another? This may be a study that is causal in nature. It can demonstrate that a group of people receiving a treatment do better on average than a group receiving a different treatment, but is not able to predict which people will do well or poorly—only the average change in risk or benefit is calculated. Leek and Peng give the example of smoking as a risk factor for lung cancer. As we all know, some people who smoke will be very badly affected, and others affected very little, but the whole group of smokers will be more likely to have lung cancer than the whole group of non-smokers (some of whom will get lung cancer too). To take a psychological example, we have the evidence that of depressed people taking antidepressants, many will do better than a matched group without the medication, but some in both groups may kill themselves.

The highest level of explanation involves a deterministic or mechanistic analysis. In this case, the evidence shows that changing one measurement is reliably and exclusively followed by a specific change in another measurement. As Leek and Peng put it, “Outside of engineering, mechanistic data analysis is extremely challenging and rarely achievable.”

When researchers are working on psychological changes in groups as a result of treatments or experiences, they may be working at anywhere from an inferential to a causal level, but their concern is still about average changes in groups. If therapists are trying to be predictive, they may not be able to do a good job of prediction if (as is common) they really do not understand how or why certain results are brought about. Without understanding what events lead to the patient chopping up Ma with the ax, predicting that event is hard; it may be right most of the time, and most weekend passes do not lead to mayhem, but the predictive failure makes it clear that the cause of the behavior is not well understood. However, the more accurate the information is—for example, the better the diagnosis—the better the chances that the individual prediction will be correct.

But human behavior rarely involves a single event that is always and exclusively followed by another specific event. Instead, a broad range of events work together to bring about most outcomes, even those that seem quite isolated, like an ax murder. In aeronautical engineering, factors like wing design can directly affect air flow, but human behavior probably has few factors that are the sole cause of an event. Predicting individual human behavior is much more like meteorology, in which a broad range of factors can determine thunderstorms (does that cold front keep moving or not?) or ax murders (does Ma take the opportunity to tell the patient who his real father is?).  Not all of these are known—or even can be known—at the time of the prediction.