Researchers test ChatGPT and other AI models on real students

Dr. William Hersh, who has taught generations of medical and clinical informatics students at Oregon Health and Science University, wondered how artificial intelligence would perform in his own classroom.

So he decided to try an experiment.

He tested six forms of extended-language generative AI models, such as ChatGPT, in an online version of his popular introductory biomedical and health informatics course to see how they performed compared to living, thinking students. A study published in the journal Digital medicine npjrevealed the answer: better than three-quarters of his human students.

“This raises concerns about cheating, but there’s a larger problem here,” Hersh said. “How do we know our students are actually acquiring and mastering the knowledge and skills they need for their future careers?”

As a professor of medical informatics and clinical epidemiology at OHSU’s School of Medicine, Hersh is particularly attuned to new technologies. The role of technology in education is not new, Hersh said, recalling his own experience as a high school student in the 1970s, during the transition from slide rules to calculators.

Yet the shift to generative AI represents an exponential leap forward.

“Clearly everyone should have some knowledge base in their field,” Hersh said. “What are the knowledge bases that people need to have in order to be able to think critically?”

Large language models

Hersh and co-author Kate Fultz Hollis, a computer scientist at OHSU, extracted knowledge assessment results from 139 students who took the 2023 Introduction to Biomedical and Health Informatics course. They created six large generative AI language models with student assessment materials from the course. According to the model, the AI ​​performed in the 50th to 75th percentile above on multiple-choice questions used in quizzes and on a final exam that required short written responses to the questions.

“The results of this study raise important questions for the future of student assessment in most, if not all, academic disciplines,” the authors write.

This study is the first to compare extended language models to students in a full-length university course in the biomedical field. Hersh and Fultz Hollis noted that a knowledge-based course like this may be particularly conducive to generative extended language models, unlike more participatory university courses that help students develop more complex skills and abilities.

Hersh recalls his experience in medical school.

“When I was a medical student, one of my attending physicians told me that I had to have all the knowledge in my head,” he said. “Even in the 1980s, that was an exaggeration. Medical knowledge has long outpaced the human brain’s ability to memorize it.”

Maintaining human contact

But he believes there is a fine line between using technical resources wisely to advance learning and relying too heavily on them to the point of inhibiting learning. Ultimately, the goal of an academic health center like OHSU is to train health professionals who can care for patients and optimize the use of data and information about them in the real world.

In this sense, he said, medicine will always need human contact.

“There are a lot of things that health professionals do that are pretty straightforward, but there are times when it gets more complicated and you have to be discerning,” he said. “That’s where it’s helpful to have a broader perspective, without necessarily having to have all the facts in your head.”

With fall classes starting soon, Hersh said he’s not worried about cheating.

“I update the course every year,” he said. “In any field of science, new advances are happening all the time, and extended language models aren’t necessarily up to date on all of those aspects. It just means we’re going to have to consider newer or more nuanced tests that you’re not going to get the answer for from ChatGPT.”