Older AI models show signs of cognitive decline, study shows

By Drew Turney
published yesterday

Older chatbots, just like people, show signs of cognitive impairment, failing on several important metrics in a test normally used on human patients.

People increasingly rely on artificial intelligence (AI) for medical diagnoses because of how quickly and efficiently these tools can spot anomalies and warning signs in medical histories, X-rays and other datasets before they become obvious to the naked eye. But a new study published Dec. 20, 2024 in the BMJ raises concerns that AI technologies like large language models (LLMs) and chatbots, like people, show signs of deteriorated cognitive abilities with age.

"These findings challenge the assumption that artificial intelligence will soon replace human doctors," the study's authors wrote in the paper, "as the cognitive impairment evident in leading chatbots may affect their reliability in medical diagnostics and undermine patients' confidence."

Scientists tested publicly available LLM-driven chatbots including OpenAI's ChatGPT, Anthropic's Sonnet and Alphabet's Gemini using the Montreal Cognitive Assessment (MoCA) test — a series of tasks neurologists use to test abilities in attention, memory, language, spatial skills and executive mental function.

MoCA is most commonly used to assess or test for the onset of cognitive impairment in conditions like Alzheimer's disease or dementia. Subjects are given tasks like drawing a specific time on a clock face, starting at 100 and repeatedly subtracting seven, remembering as many words as possible from a spoken list, and so on. In humans, 26 out of 30 is considered a passing score (ie the subject has no cognitive impairment.

More:
https://www.livescience.com/technology/artificial-intelligence/older-ai-models-show-signs-of-cognitive-decline-study-shows