An EPFL study published in the American journal PNAS tested systems like ChatGPT to respond to university assessments. The AI can provide 85% correct answers, if it is informed beforehand. If the study were re-run today, this figure would be even higher.
Focusing on GPT-3.5 and GPT-4, the scientists used eight different strategies to produce responses. They found that GPT-4 answered 65.8% of questions correctly on average and could even provide the correct answer in at least one strategy for 85.1% of questions.
More precisely, 65% of correct answers were obtained using the most basic strategy, without prior knowledge. “With some knowledge of the subject, it was possible to achieve a success rate of 85%,” says Anna Sotnikova, co-author of the article.
>> Read also: Universities are gradually putting in place rules of the game in the face of AI
Researchers “shocked” by these results
The 85% correct answer rate “was really a shock,” admits Anna Sotnikova. “We were surprised by the results. No one expected that the AI assistants would obtain such a high percentage of correct answers in so many courses,” continues the researcher.
AI assistants will not get worse, they will only get better. The scientists’ conclusion is that if the study were restarted today, the numbers would be even higher.
50 courses studied
In their study, scientists from the Faculty of Computer Science and Communications looked at 50 EPFL courses in order to measure the current performance of these large language models (LLM) in higher education course evaluations. The courses that were selected are part of nine online Bachelor’s and Master’s programs and cover a wide range of disciplines, including computer science, mathematics, biology, chemistry, physics and materials science.
“This data was brought together in a format that we believed would most closely resemble the way students would communicate this information to models,” explains Antoine Bosselut, assistant professor and member of the EPFL AI Center, quoted in the press release. .
Adapt education
“In the short term, we should insist that assessments be more difficult, not in the sense of the difficulty of the questions, but in the sense of the complexity of the assessment itself,” suggests Antoine Bosselut. In the longer term, it is clear that education systems will have to adapt, according to researchers.
“It’s only the beginning and I think an analogy can be made between current LLMs and calculators. When they were introduced there were the same concerns that children would no longer learn the mathematics”, notes Beatriz Borges, co-author of this research.
“Today, in the early stages of education, calculators are generally not allowed, but from higher grades they are present, to perform lower level tasks while students acquire more advanced skills” , concludes the researcher.
lia with ats
