AI-generated college admissions essays tend to appear masculine and privileged, study finds

In a review of thousands of human-written and AI-generated college admissions essays, researchers found that AI-generated essays are most similar to essays written by male students, with higher socioeconomic status and higher levels of social privilege. AI-generated writing is also less varied than that written by humans.

“We wanted to find out what these patterns look like that we see in essays written by humans in a ChatGPT world,” said AJ Alvero, assistant research professor in the Department of Information Sciences at Cornell Ann S. Bowers College of Computing and Information. Science. “If there is a strong connection between human writing and identity, how does this compare to essays written by AI?”

Rene Kizilcec, associate professor of information sciences at Cornell Bowers CIS, is co-author of “Large Language Models, Social Demography, and Hegemony: Comparing Authorship in Human and Synthetic Text,” published September 27 in the journal Journal of Big Data.

This research stems from Alvero’s dissertation at Stanford University. Part of his research involved an analysis of approximately 800,000 college admissions essays written between 2015 and 2017 by prospective students in the University of California system.

“We consistently found that there was a strong connection between applicants’ profiles (their test scores, their demographic information, even the high schools they were applying to) and their admissions essays,” Alvero said. “The relationship was so strong that we were consistently able to predict an applicant’s SAT score, to within about 120 points.”

“The way we speak can encode and contain information about our past and who we are,” he said, “and it’s very similar in writing, at least with personal statements.”

For this work, Alvero and the team compared the writing style of more than 150,000 college admissions essays, submitted to both the University of California system and an engineering program d An elite private university on the East Coast, with a pool of over 25,000 essays generated with GPT-3.5 and GPT-4 are asked to answer the same essay questions as human test takers.

For their analysis, the researchers used the Linguistic Inquiry and Word Count, a program developed in the mid-1990s by social psychologist James W. Pennebaker of the University of Texas, which counts the frequencies of writing features, such as punctuation and use of pronouns, and crossovers. references these counts with an external dictionary.

“One of the first big data analyzes of college admissions essays was done about ten years ago by Pennebaker,” said Alvero, “and we wanted to try to build a solid understanding of these patterns in the establishments and over time, and we did it using the same method they used.

Alvero, Kizilcec and team found that although LLM writing styles do not represent any particular group in social comparison analyses, they “sound”, in terms of word selection and usage, more like male students who came from more privileged places and backgrounds.

For example, AI has been found to use longer words (six letters or more) on average than human writers. Additionally, AI-generated writing tends to be less varied than human-written essays, even though it is more similar to essays written by private school applicants than to those by private school students. public.

Additionally, humans and AI tend to write about their affiliations (with groups, people, organizations, and friends) at similar rates, although the AI ​​actually has no affiliations. As LLMs like ChatGPT become more popular and more refined, they will be used in all kinds of contexts, including college admissions.

“It’s likely that students will use AI to help them write these essays — probably not by asking it to write everything, but rather by asking it for help and feedback,” Kizilcec said. “But even then, the suggestions that these models will make might not be well aligned with the values, the type of linguistic style, that would be an authentic expression of these students.

“It’s important to remember that if you use an AI to help you write an essay, it will probably look less like you and more like something fairly generic,” he said. “And students should know that for those reading these essays, it won’t be too difficult for them to determine who has used AI extensively. The key will be to use it to help students tell their own stories and improve what they think.” they want to transmit, not replace their own voice.

Alvero and Anthony Lising Antonio, associate professor of education at the Graduate School of Education at Stanford University, are co-corresponding authors.