Could ChatGPT get an Engineering Degree? Evaluating Higher Education
Vulnerability to AI Assistants
Beatriz Borges, Negar Foroutan, Deniz Bayazit, Anna Sotnikova, Syrielle Montariol, Tanya Nazaretzky, Mohammadreza Banaei, Alireza Sakhaeirad, Philippe Servant, Seyed Parsa Neshaei, Jibril Frej, Angelika Romanou, Gail Weiss, Sepideh Mamooler, Zeming Chen, Simin Fan, Silin Gao, Mete Ismayilzada, Debjit Paul, Alexandre Sch\"opfer, Andrej Janchevski, Anja Tiede, Clarence Linden, Emanuele Troiani, Francesco Salvi, Freya Behrens, Giacomo Orsi, Giovanni Piccioli, Hadrien Sevel, Louis Coulon, Manuela Pineros-Rodriguez, Marin Bonnassies, Pierre Hellich, Puck van Gerwen, Sankalp Gambhir, Solal Pirelli, Thomas Blanchard, Timoth\'ee Callens, Toni Abi Aoun, Yannick Calvino Alonso, Yuri Cho, Alberto Chiappa, Antonio Sclocchi, \'Etienne Bruno, Florian Hofhammer, Gabriel Pescia, Geovani Rizk, Leello Dadi, Lucas Stoffl, Manoel Horta Ribeiro, Matthieu Bovel, Yueyang Pan, Aleksandra Radenovic, Alexandre Alahi, Alexander Mathis, Anne-Florence Bitbol, Boi Faltings, C\'ecile H\'ebert, Devis Tuia, Fran\c{c}ois Mar\'echal, George Candea, Giuseppe Carleo, Jean-C\'edric Chappelier, Nicolas Flammarion, Jean-Marie F\"urbringer, Jean-Philippe Pellet, Karl Aberer, Lenka Zdeborov\'a, Marcel Salath\'e, Martin Jaggi, Martin Rajman, Mathias Payer, Matthieu Wyart, Michael Gastpar, Michele Ceriotti, Ola Svensson, Olivier L\'ev\^eque, Paolo Ienne, Rachid Guerraoui, Robert West, Sanidhya Kashyap, Valerio Piazza, Viesturs Simanis, Viktor Kuncak, Volkan Cevher, Philippe Schwaller, Sacha Friedli, Patrick Jermann, Tanja K\"aser, Antoine Bosselut
AI assistants are being increasingly used by students enrolled in higher
education institutions. While these tools provide opportunities for improved
teaching and education, they also pose significant challenges for assessment
and learning outcomes. We conceptualize these challenges through the lens of
vulnerability, the potential for university assessments and learning outcomes
to be impacted by student use of generative AI. We investigate the potential
scale of this vulnerability by measuring the degree to which AI assistants can
complete assessment questions in standard university-level STEM courses.
Specifically, we compile a novel dataset of textual assessment questions from
50 courses at EPFL and evaluate whether two AI assistants, GPT-3.5 and GPT-4
can adequately answer these questions. We use eight prompting strategies to
produce responses and find that GPT-4 answers an average of 65.8% of questions
correctly, and can even produce the correct answer across at least one
prompting strategy for 85.1% of questions. When grouping courses in our dataset
by degree program, these systems already pass non-project assessments of large
numbers of core courses in various degree programs, posing risks to higher
education accreditation that will be amplified as these models improve. Our
results call for revising program-level assessment design in higher education
in light of advances in generative AI.
These charts contain model-linked results. The charts
above also include unlinked source profiles. Compare
scores within each evaluation. Model quality, usage,
transparency and system performance measure different
things.