Available from the end of 2022, ChatGPT has surpassed the 100 million user mark in less than a month. Since then, AI has gradually become established in many homes and businesses. However, until now, there has been no in-depth study of the potential impact of AI assistants on the assessment methods used by educational institutions.
EPFL researchers therefore carried out a large-scale study on 50 courses at their school. They are part of nine online Bachelor’s and Master’s programs and cover a wide range of disciplines. “This data was brought together in a format that we thought would most closely resemble the way students would communicate it,” explains Antoine Bosselut, head of the Natural Language Processing Laboratory (NLP) and member of the AI Center at the University. EPFL.
The scientists used eight prompting strategies (the art of formulating requests in such a way that artificial intelligence fully understands what you are looking for) to produce answers. They found that GPT-4 correctly answers 65.8% of questions on average and can even provide the correct answer in at least one prompting strategy for 85.1% of questions. “We were surprised by the results. No one expected that AI assistants would get such a high percentage of correct answers in so many courses,” says Anna Sotnikova, scientist at NLP and co-author of the publication.
The researchers have thus theoretically demonstrated the vulnerability of assessments to the use of AI by students. If ChatGPT is able to obtain its engineering degree just by digesting the courses as the students receive them, what proves to us that they will actually have the required academic skills? “In the short term, we should insist that assessments be more difficult, not in the sense of the difficulty of the questions, but in the sense of the complexity of the assessment itself,” suggests Antoine Bosselut, in the press release published this Friday.
