Developing revolutionary AI tools for healthcare

Reinforcement Learning, an artificial intelligence approach, has the potential to help physicians design sequential treatment strategies to improve patient outcomes. However, it requires significant improvement before clinical application, according to a recent study by researchers at Weill Cornell Hospital and Rockefeller University.

Reinforcement Learning (RL) is a category of machine learning algorithms capable of making a series of decisions over time. Responsible for recent advances in AI, including superhuman performance in chess and Go, the RL can analyze evolving patient conditions, test results and previous responses to treatments to suggest the best next step in personalized care. This approach is particularly promising for decision-making in the management of chronic or psychiatric illnesses.

This research, published in the Proceedings of the Neural Information Processing Systems (NeurIPS) Conference and presented on December 13, presents “Episodes of Care” (EpiCare), the first RL benchmark for the healthcare sector .

“Benchmarks have led to improvements in many machine learning applications, such as computer vision, natural language processing, and speech recognition. We hope they will now help advance RL in healthcare,” said Dr. Logan Grosenick, assistant professor of psychiatric neuroscience and director of this research.

RL agents refine their actions based on the feedback they receive, gradually learning a policy that improves their decision-making. “However, our results show that although current methods are promising, they are extremely data intensive,” adds Dr. Grosenick.

The researchers first tested the performance of five state-of-the-art online RL models on EpiCare. All five models outperformed a standard of care, but only after being trained on thousands, if not tens of thousands, of simulated treatment episodes. In reality, RL methods would never be directly trained on patients, which led researchers to then evaluate five “off-policy” evaluation (OPE) methods: popular approaches aimed at using historical data ( from clinical trials, for example) to avoid the need to collect data online. Using EpiCare, they found that even the most advanced OPE methods failed to provide accurate performance for health data.

“Our results indicate that current OPE methods cannot be considered reliable for accurately predicting reinforcement learning performance in longitudinal health scenarios,” said Dr. Mason Hargrave, first author and researcher at the University. Rockefeller. This finding highlights the importance of developing more precise benchmarking tools, such as EpiCare, to evaluate existing RL approaches and provide metrics measuring improvement.

“We hope that this work will facilitate more reliable evaluation of reinforcement learning in healthcare environments and help accelerate the development of better training algorithms and protocols suitable for medical applications,” concluded Dr. Grosenick.

Adapting Convolutional Neural Networks to Interpret Graphical Data

In a second publication presented the same day at NeurIPS, Dr. Grosenick shared his research on adapting convolutional neural networks (CNNs), widely used to process images, to make them efficient for data structured in graphs, such as brain, genetic or protein networks. The success of CNNs in image recognition tasks in the early 2010s laid the foundation for deep learning and the modern era of neural network-based AI applications.

“We are often interested in analyzing brain imaging data that looks more like graphs, with vertices and edges, than images. But we realized that there was nothing truly equivalent to CNNs for graph-structured data,” said Dr. Grosenick.

Brain networks are typically represented as graphs where brain regions (represented as vertices) transmit information to other brain regions along “edges” that connect them and represent the strength of communication between them. This is also true for genetic and protein networks, human and animal behavioral data, and the geometry of chemical compounds like drugs. By directly analyzing these graphs, we can more accurately model dependencies and patterns between local and more distant connections.

Isaac Osafo Nkansah, a research associate who was in Grosenick’s lab at the time of the study and first author of the paper, contributed to the development of the Quantized Graph Convolution Networks (QuantNets) framework, which generalizes CNNs to graphs. “We are now using it to model EEG (brain electrical activity) data in patients. We can use a network of 256 sensors distributed across the scalp, taking measurements of neuronal activity — it’s a graph,” explained Dr. Grosenick. “We reduce these large graphs into more interpretable components to better understand how dynamic brain connectivity changes as patients are treated for depression or obsessive-compulsive disorder.”

Researchers envision widespread applications for QuantNets. For example, they also seek to model graph-structured pose data to track behavior in mouse models and in human facial expressions extracted using computer vision.

“While we are still navigating the safety and complexity of applying cutting-edge AI methods to patient care, each advancement, whether a new benchmarking framework or a more precise model , is slowly moving us closer to personalized treatment strategies with the potential to significantly improve patient outcomes,” concluded Dr. Grosenick.