San Diego, United States – The artificial intelligence (AI) tool ChatGPTwhich can provide information about headaches, makes significant mistakes when it comes to drug treatments for migraine, but does better when it comes to non-pharmacological therapies, according to an assessment by headache specialists of head.
The results, presented on June 13 during the American Headache Society (AHS) 2024 Annual Meeting suggest that doctors should advise patients not to rely solely on ChatGPT for accurate information about migraine treatments, at least not in its current version, he told Medscape Medical News the searcher Robert Bonakdardirector of pain management at the Scripps Center for Integrative Medicine in San Diego.
The good news, however, is that more precise AI alternatives may be in the works, Robert Bonakdar added.
Can I trust AI?
Clinicians have mixed feelings about AI, including sophisticated language models like ChatGPT. They are intrigued and are venturing into it, but are aware of its limitations and outdated information, said Bonakdar, who has treated migraines for more than two decades and is experienced in using ChatGPT.
Although there are other AI tools, ChatGPT is one of the most comprehensive and is the one headache specialists are most likely to use, Bonakdar said. Studies show the tool is useful in some areas of medicine, such as orthopedics, but less well for other areas, such as neurosurgery, he added.
“The question is whether AI can be trusted for migraine,” he added.
For this study, researchers asked ChatGPT 14 common questions related to migraine treatment options. Questions focused on the effectiveness and safety of conventional treatments (preventative medications, acute treatments, acute non-oral therapies such as patches and sprays and procedures involving nerve blocks) and non-pharmacological treatments (diet/supplements, exercise, stimulation devices, physical therapies such as massage, self-care and behavioral therapies).
The researchers then presented the AI’s responses to four headache specialists: a neurologist, a neurology assistant physician, a United Council for Neurologic Subspecialties board-certified physician, and a headache researcher.
These experts rated the ChatGPT responses on a Likert scale from 1 (strongly disagree) to 5 (strongly agree) in terms of completeness, accuracy, and usefulness.
Participants gave higher overall ratings for non-pharmaceutical treatments. Their highest rating – for behavioral therapies – was 4.75, while their lowest rating – for diets and supplements – was 4.
“ChatGPT has performed quite well in advising on non-pharmacological therapies, but it has not been perfect,” commented Dr. Bonakdar.
For pharmacological treatments, the scores are lower. The highest score was 3.95 (for procedures) and the lowest was 3.5 (for preventative medications).
AI errors here were “much more pronounced” than for non-drug interventions, Bonakdar said.
Inappropriate options
Reviewers noted that the ChatGPT also recommended inappropriate treatment options, such as opioids and migraine surgery.
They also noticed that the tool incorrectly stated that the Cefaly ® device required a prescription when external trigeminal nerve stimulation is available over the counter. ChatGPT also failed to note the potential health risks of migraine treatments during pregnancy, including feverfew, an herb containing compounds believed to help treat or prevent inflammatory conditions.
Headache specialists also pointed out that the AI tool did not mention approved drugs such as Nurtec ® (rimegepant) and Qulipta ® (atogepant), two calcitonin gene-related peptide receptor blockers , nor did it include important safety information such as side effects of valproic acid, the anticonvulsant Topamax (topiramate), and beta-blockers.
ChatGPT is “a good starting point” for general information about migraine, but because the information isn’t personalized, it’s important for patients to discuss treatments with their clinician, Bonakdar said.
“ChatGPT doesn’t know you. It doesn’t know your medications, your contraindications or your allergies,” he added.
In an editorial commented for Medscape Medical Newsthe Dre Nina Rigginspresident of the Brain Performance Center and Research Institute and director of the Headache Center at the Neuron Clinic in San Diego, USA, said the study provided important information, but she would have liked to see further evaluation of ChatGPT’s diagnostic capabilities.
Dr. Riggins stressed the importance of understanding the capabilities and limitations of AI in the medical field. We are aware of the “hallucinations” of AI, when it does not provide correct information, but we also understand that the use of technology can be extremely important in medicine, even life-saving.
Many inaccuracies related to ChatGPT software can be attributed to outdated information, she added. Understanding how this AI tool uses this information could “provide specific avenues for improving this technology,” Dr. Riggins said.
In the future, AI could help triage headache patients, read imaging reports, and improve overall access to care. Additionally, by reducing the amount of paperwork doctors have to deal with, AI could allow doctors to spend more time with their patients, she added.
In the meantime, Dr. Riggins agreed that clinicians should advise patients not to learn about the effectiveness and safety of treatments from ChatGPT.
A more targeted AI tool
Dr. Bonakdar also presented the results of a second study involving a more targeted AI assistant that the researchers created with Wellkasa, a company that studies AI technology and evidence-based lifestyle medicine. The group trained the AI model, known as My AI Assistant (Maia) to systematically provide the most precise answers possible.
“For example, if Maia answered that Cefaly ® required a prescription, which is false, she got a low score, and if she answered correctly that it did not, she got a high score,” said Dr. Bonakdar.
Researchers asked 23 migraine patients to try Maia for a week. Patients were asked to ask Maia questions related to migraine, and could do so in a chat room or over the phone.
The study found that knowledge and confidence scores on migraine prevention, treatment and lifestyle improved for 11 of 12 questions (91.7%).
“These results are critical because we know that education alone can improve migraine outcomes and medication benefits,” Bonakdar said.
While education can be time-consuming and not always personalized, “with Maia, we have discovered that migraine education can be simplified because learning about migraine becomes a very personal learning path through simple, natural conversations,” he added.
The study also showed statistically significant improvements on 4 of 12 points relating to medication management, side effects of therapies, health and diet aspects and multimodal approaches to migraine management.
Participants with less migraine-related disability were significantly more satisfied with Maia than those with greater migraine-related disability (p
In their feedback, study participants said the tool answered questions they don’t want to bother their doctor with and saved them from having to do multiple searches for migraine medications. However, some said they wished Maia could answer questions about comorbidities and track the progress of their migraines.
While AI is seen by some as having the potential to replace clinicians, it is more likely that patients will use the information they glean from AI to address with their doctors, particularly in areas such as interventions health and diet, which can be quite nuanced, said Robert Bonakdar.
Dr. Riggins said she was particularly impressed by the helpful advice Maia provided on lifestyle changes such as sleep and diet.
All AI tools, including ChatGPT and Maia, should ultimately make migraine education safer for both patients and physicians, Dr. Riggins said.
Link of interest
Drs Bonakdar and Riggins have declared no relevant conflicts of interest.
This article was translated from Medscape.com using multiple editorial tools, including AI, in the process. The content was reviewed by the editorial staff before publication.
