A study published in a famous medical journal reveals the results of a medical test taken by chatbots like ChatGPT, Gemini and Claude. The first tests reveal disorders frequently observed in patients with dementia.
Do chatbots need to see a psychiatrist? A study, published in the Christmas edition of BMJ (British Medical Journal) on December 18, 2024, reveals that artificial intelligence tools show signs comparable to mild cognitive impairment. These disorders are similar to those observed in patients with the early stages of dementia. These findings come from a standard test among researchers, the MoCA (Montreal Cognitive Assessment), commonly used to detect problems with memory, attention and reasoning in older adults.
During this exercise, the chatbots showed variable performance. For example, the ChatGPT 4o version obtained 26 points out of 30. Barely more than the latest version of Claude, with 25 out of 30, a score considered normal. Others, like Gemini, capped at 16 points, a result revealing significant shortcomings.
What is striking is their difficulty in responding in certain specific categories: chatbots fail to perform visuospatial tasks (such as drawing a clock showing a given time) or executive exercises (such as connecting numbers and letters in order). . The researchers note that these failures are comparable to the impairments observed in people with cognitive disorders. They describe these behaviors as “ digital dementia », We can read in the study.
ChatGPT and Gemini will not replace your psychologist or doctor
The comparison with human dementia relies on the way these models process information. Much like a failing brain, chatbots can appear competent at simple tasks, like naming objects or answering direct questions.
However, when it comes to abstraction or planning, they show glaring weaknesses. For example, only ChatGPT 4o passed a test involving complex interference, where the written word and its color do not match. These results reveal an inability to manage conflicting or visually complex information, two essential skills for humans.
Despite their limitations, chatbots remain impressive in specific areas. They excel at text-based diagnostics or quick responses. Nevertheless, the researchers point out that their inability to handle certain fundamental tasks shows that they are far from replacing doctors, particularly in fields like neurology or psychology. In other words, although these models imitate human language well, they lack the deep understanding needed to accomplish more abstract or complex tasks.
This study also highlights an unexpected point: these models could themselves be seen as “virtual patients”, with doctors being monitored based on their improvements.
