Systems like ChatGPT-4 can correctly answer 85% of college assessment questions. These results, published in the American journal PNAS, constituted “a shock” for the authors of the study.
ChatGPT arrived on the public scene in late 2022, attracting over 100 million users in its first month. In higher education, the student community is increasingly using this type of AI (artificial intelligence) assistant, the Federal Polytechnic School of Lausanne (EPFL) said in a press release on Friday.
In their study, scientists from the Faculty of Computer Science and Communications looked at 50 EPFL courses in order to measure the current performance of these large language models (LLM) in higher education course evaluations.
The courses that were selected are part of nine online Bachelor’s and Master’s programs and cover a wide range of disciplines, including computer science, mathematics, biology, chemistry, physics and materials science.
“This data was brought together in a format that we believed would most closely resemble the way students would communicate this information to models,” explains Antoine Bosselut, assistant professor and member of the EPFL AI Center, quoted in the press release.
Focusing on GPT-3.5 and GPT-4, the scientists used eight different strategies to produce responses. They found that GPT-4 correctly answers an average of 65.8% of questions and can even provide the correct answer in at least one strategy for 85.1% of questions.
“A shock”
“We were surprised by the results. No one expected that AI assistants would get such a high percentage of correct answers in so many courses,” says Anna Sotnikova, co-author of the article.
Indeed, 65% of correct answers were obtained using the most basic strategy, without prior knowledge. “With a certain knowledge of the subject, it was possible to achieve a success rate of 85%, which was really a shock,” adds the researcher.
However, these AI assistants will not get worse, they will only improve. The scientists’ conclusion is that if the study were restarted today, the numbers would be even higher.
Adapt education
“In the short term, we should insist that assessments be more difficult, not in the sense of the difficulty of the questions, but in the sense of the complexity of the assessment itself,” suggests Antoine Bosselut. In the longer term, it is clear that education systems will have to adapt.
“It’s only the beginning and I think an analogy can be made between current LLMs and calculators. When they were introduced there were the same concerns that children would no longer learn the mathematics”, notes Beatriz Borges, co-author of this research.
“Today, in the early stages of education, calculators are generally not allowed, but from higher grades they are present, to perform lower level tasks while students acquire more advanced skills” , concludes the researcher.
This article was automatically published. Source: ats
Related News :