Can machines spot faces in inanimate objects?

In 1994, Florida jewelry designer Diana Duyser discovered what she thought was the image of the Virgin Mary in a grilled cheese sandwich, which she saved and then sold at auction for $28,000. But what do we really know about pareidolia, the phenomenon of seeing faces and patterns in objects when they’re not really there?

A new study from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) delves into this phenomenon, introducing a large human-labeled dataset of 5,000 pareid images, far surpassing previous collections. Using this dataset, the team discovered several surprising results about the differences between human and machine perception, and how the ability to see faces on a slice of toast could have saved lives of your distant loved ones.

The study is published on the arXiv preprint server.

“Facial pareidolia has long fascinated psychologists, but it is largely unexplored in the computer vision community,” says MIT’s Mark Hamilton, Ph.D. student in electrical and computer engineering, affiliated with CSAIL and principal investigator of the work. “We wanted to create a resource that could help us understand how humans and AI systems process these illusory faces.”

So what do all these fake faces reveal? For one thing, AI models don’t seem to recognize pareid faces like we do. Surprisingly, the team found that only after training algorithms to recognize animal faces did they become significantly better at detecting pareid faces. This unexpected connection hints at a possible evolutionary link between our ability to spot animal faces – crucial for survival – and our tendency to see faces in inanimate objects.

“A result like this seems to suggest that pareidolia might not come from human social behavior, but from something deeper: like quickly spotting a hiding tiger or identifying which direction a deer is looking so that our primordial ancestors can hunt,” says Hamilton.

Another intriguing finding is what researchers call the “Goldilocks zone of pareidolia,” a class of images where pareidolia is most likely to occur.

“There is a specific range of visual complexity in which humans and machines are more likely to perceive faces in objects other than faces,” says William T. Freeman, professor of electrical engineering and computer science at MIT and researcher principal of the project. “Too simple and there isn’t enough detail to form a face. Too complex and it becomes visual noise.”

To discover this, the team developed an equation that models how people and algorithms detect illusory faces. Analyzing this equation, they found a clear “pareidolic peak” where the probability of seeing faces is highest, corresponding to images that have “just the right amount” of complexity. This predicted “Goldilocks zone” was then validated through testing with real human subjects and AI face detection systems.

This new dataset, “Faces in Things,” dwarfs those from previous studies that typically used only 20 to 30 stimuli. This scale allowed the researchers to explore the behavior of state-of-the-art face detection algorithms after fine-tuning on pareidolic faces, showing that not only could these algorithms be modified to detect these faces, but they could also act as a silicon. replaces our own brains, allowing the team to ask and answer questions about the origins of pareidolic face detection that are impossible to ask in humans.

To create this dataset, the team selected approximately 20,000 candidate images from the LAION-5B dataset, which were then meticulously labeled and judged by human annotators. This process involved drawing bounding boxes around perceived faces and answering detailed questions about each face, such as perceived emotion, age, and whether the face was accidental or intentional.

“Collecting and annotating thousands of images was a monumental task,” says Hamilton. “Much of the dataset owes its existence to my mother,” a retired banker, “who spent countless hours lovingly labeling the images for our analysis.”

The study also has potential applications in improving face detection systems by reducing false positives, which could have implications in areas such as self-driving cars, human-machine interaction and robotics. The dataset and models could also help in areas such as product design, where understanding and controlling pareidolia could create better products.

“Imagine being able to automatically change the design of a car or a child’s toy to make it more user-friendly, or ensuring that a medical device doesn’t appear inadvertently threatening,” says Hamilton.

“It’s fascinating how humans instinctively interpret inanimate objects with human-like traits. For example, when you look at an electrical outlet, you can immediately imagine it singing, and you can even imagine how it would “move “However, algorithms don’t naturally recognize these caricatured faces in the same way that we do,” says Hamilton.

“This raises intriguing questions: What explains this difference between human perception and algorithmic interpretation? Is pareidolia beneficial or detrimental? Why don’t algorithms experience this effect like we do? These questions triggered our investigation, as this classic psychological phenomenon in humans has not been explored thoroughly in algorithms.

As researchers prepare to share their data with the scientific community, they are already looking toward the future. Future work could involve training visual language models to understand and describe pareid faces, which could lead to AI systems capable of interacting with visual stimuli in more human-like ways.

“This is a delightful article! It’s fun to read and makes me think. Hamilton et al. propose a tantalizing question: why do we see faces in things?” said Pietro Perona, the Allen E. Puckett Professor of Electrical Engineering at Caltech, who was not involved in the work.

“As they point out, learning from examples, especially animal faces, only goes half of the way to explaining the phenomenon. I bet thinking about this question will teach us something important about how our visual system generalizes beyond the training it receives throughout life.

Hamilton and Freeman’s co-authors include Simon Stent, a research scientist at the Toyota Research Institute; Ruth Rosenholtz, senior research scientist in the Department of Brain and Cognitive Sciences, NVIDIA research scientist and former CSAIL member; and CSAIL-affiliated postdocs Vasha DuTell, Anne Harrington MEng ’23, and research scientist Jennifer Corbett.

This work is being presented this week at the European Conference on Computer Vision.

This story is republished courtesy of MIT News (web.mit.edu/newsoffice/), a popular site that covers news in MIT research, innovation and education.