Study examines difference between human and machine capabilities

A police officer hears an eyewitness account of a car accident, a doctor diagnoses an illness based on symptoms listed by a patient, or a friend recommends a restaurant. The memories of others are a source of information for us that influences how we act. Cognitive neuroscientists at Ben-Gurion University of the Negev studied how humans assess the reliability of these memories and the difference between human and machine capabilities.

The results were recently published in the journal Proceedings of the National Academy of Sciences.

Years of research indicate that our memories are prone to forgetting and distortion. People’s memories are not accurate descriptions of the past, but are prone to errors, even after short periods of time. The information from these memories is important to us because much of our knowledge of the world is based on information from other people’s memories. So how can we base information on memory that is not always reliable?

Dr. Talya Sadeh from the Department of Cognitive and Brain Sciences at Ben-Gurion University of the Negev decided to look into this question and understand how humans can recognize and verify that other people’s memories are accurate.

“Many people have knowledge that comes from sharing episodic memories, knowledge that we use to make decisions, form opinions, and so on,” she noted. “My research has looked at how we come to base our sometimes very important knowledge on memories that are not always reliable. Can natural language processing models, such as those we are all familiar with (e.g., ChatGPT and others), help us identify the truth of memory?”

To do this, she conducted a study that simulated real-life situations in which a person is asked to judge whether memories told by another person are true or not. For example: “I remember that a woman was at a party because I remember that she arrived late and that she was wearing a very beautiful dress” or “I remember that the car did not stop at the red light because I noticed the speed before it reached the intersection, as the light changed from green to yellow.”

Participants were asked to directly judge whether the memories they had recounted were true or not, based on these descriptions. They were then asked to rate the quality of the other’s memory by giving a quantitative rating on questions such as the vividness and level of detail of the memory and the degree of confidence the interlocutor had in it.

The comparison with a machine learner was made based on the words in the memory descriptions that were most indicative of correct (or incorrect) memory, between humans and the machine learning model. Of the 20 words that best indicated memory accuracy, 14 were shared by humans and machines. Therefore, the results indicate that humans have the ability to directly assess others’ memories and determine whether they are true or false, and they do so based on much of the same information as a language model (and as well as the model).

However, the reliability of others’ memories can be predicted even better (within 10%) if, instead of relying on a direct assessment of memory reliability, we ask raters to judge the qualities of the memory: the extent to which they think it is associated with a rich, vivid, and detailed sensory experience, and the extent to which the person sharing the memory appears confident.

While a language model is based on extracting statistical rules, people’s decisions depend on their sensitivity to information indicating the qualities of memory. This allows humans to evaluate other people’s memories very well, and perhaps even better than a language model trained to extract statistical rules from texts describing people’s memories.

Since we use language to share mental states, thoughts, and beliefs, this ongoing learning of language serves as a tool for us to validate the memories and experiences of others. This study is an important step in understanding how humans think about memory sharing and its importance in creating social knowledge.

“Humans have the ability to take advantage of their status as social beings to learn quickly from others. Much of their knowledge comes from the fact that we share real-life experiences with each other, and we have successfully shown that the machine cannot yet take our place in terms of personal memory,” concludes Dr. Sadeh.

The research group included Avi Gamran of the Department of Psychology and Lilach Lieberman of the Department of Cognitive and Brain Sciences at Ben-Gurion University of the Negev, Professor Ian Dobbins of the University of Washington in Missouri, and Professor Michael Gilead of Tel Aviv University.