Current models have limited effectiveness in predicting treatments, study finds

The quest for personalized medicine, a medical approach in which practitioners use a patient’s unique genetic profile to tailor individual treatment, has become a critical goal in the healthcare industry. But a new study led by Yale shows that the mathematical models currently available for predicting treatments have limited effectiveness.

In an analysis of clinical trials of several schizophrenia treatments, researchers found that mathematical algorithms were able to predict patient outcomes in the specific trials for which they were developed, but did not work for patients participating in different trials.

The work is published in the journal Science.

“This study really challenges the status quo of algorithm development and raises the bar for the future,” said Adam Chekroud, adjunct assistant professor of psychiatry at the Yale School of Medicine and corresponding author of the paper. “Right now, I would say we need to see the algorithms work in at least two different contexts before we can really get excited.”

“I’m always optimistic,” he added, “but as medical researchers, we have some serious things to solve.”

Chekroud is also president and co-founder of Spring Health, a private company that provides mental health services.

Schizophrenia, a complex brain disorder that affects about 1 percent of the U.S. population, perfectly illustrates the need for more personalized treatments, researchers say. Up to 50% of patients diagnosed with schizophrenia do not respond to the first antipsychotic medication prescribed, but it is impossible to predict which patients will respond to therapies and which will not.

Researchers hope that new technologies using machine learning and artificial intelligence can produce algorithms that better predict which treatments will work for different patients, and help improve outcomes and reduce costs of care.

However, due to the high cost of running a clinical trial, most algorithms are only developed and tested using a single clinical trial. But the researchers hoped that these algorithms would work if tested on patients with similar profiles and receiving similar treatments.

For the new study, Chekroud and his colleagues at Yale wanted to see if this hope was really true. To do this, they pooled data from five clinical trials on schizophrenia treatments made available through the Yale Open Data Access (YODA) project, which advocates and supports the responsible sharing of clinical research data. In most cases, they found, the algorithms effectively predicted patient outcomes for the clinical trial in which they were developed. However, they failed to effectively predict outcomes for schizophrenia patients treated in different clinical trials.

“The algorithms almost always worked the first time,” Chekroud said. “But when we tested them on patients in other trials, the predictive value was no better than chance.”

The problem, Chekroud says, is that most of the mathematical algorithms used by medical researchers were designed for use on much larger data sets. Clinical trials are expensive and time-consuming to conduct, so studies typically enroll fewer than 1,000 patients. Applying powerful AI tools to analyzing these smaller data sets, he said, can often result in “overfitting,” in which a model has learned response patterns that are idiosyncratic, or specific only to the data in the initial trial, but which disappear when additional new data are included.

“The reality is we need to think about algorithm development the same way we think about new drug development,” he said. “We need to see the algorithms working at several different times or contexts before we can really believe in them.”

In the future, the inclusion of other environmental variables may or may not improve the success of algorithms in analyzing clinical trial data, the researchers added. For example, does the patient abuse drugs or have personal support from family or friends? These are the kinds of factors that can affect treatment results.

Most clinical trials use specific criteria to improve the chances of success, such as guidelines for which patients should be included (or excluded), careful measurement of outcomes, and limits on the number of doctors administering treatments. Real-world settings, meanwhile, feature a greater variety of patients and greater variation in treatment quality and consistency, the researchers say.

“In theory, clinical trials should be the easiest place for algorithms to work. But if algorithms cannot generalize from one clinical trial to another, it will be even more difficult to use them in clinical practice” , said co-author John. Krystal, Robert L. McNeil, Jr. Professor of Translational Research and Professor of Psychiatry, Neuroscience, and Psychology at the Yale School of Medicine. Krystal is also chair of the Department of Psychiatry at Yale.

Chekroud suggests that increased efforts to share data among researchers and the banking of additional data by healthcare providers on a large scale could help increase the reliability and accuracy of AI-based algorithms.

“Although the study focused on schizophrenia trials, it raises difficult questions for personalized medicine in general and its application in cardiovascular disease and cancer,” said Philip Corlett, associate professor of psychiatry at Yale. and co-author of the study.

Other authors of the study at Yale are Hieronimus Loho; Ralitza Gueorguieva, senior research scientist at the Yale School of Public Health; and Harlan M. Krumholz, Harold H. Hines Jr. Professor of Medicine (Cardiology) at Yale.