Researchers test new “mixed reality” headsets

Among the hottest consumer technologies today are “mixed reality” or “spatial computing” headsets that convincingly blend real-world views with digital content.

A key technology behind these gadgets is video passthrough, which involves blocking all light, so users must rely on the headsets’ cameras to see the outside world around them via real-time video streamed to small screens. This arrangement allows users to physically interact with their surroundings and go about their daily activities, while viewing additional digital content, ranging from familiar device applications to innovative gaming scenarios. If the tech companies’ visions come true, users would wear these headsets for extended periods of time, even all day at work and at home, paving the way for new modes of human-machine and social interaction.

To put video passthrough to the test, a diverse team of Stanford researchers recently conducted field tests as well as longitudinal analyzes of their personal journeys and interpersonal interactions. As described in a new study in Technology, mind and behavior, overall user experiences were – understandably – mixed, with moments of both wonder and instability. The researchers therefore recommend caution regarding prolonged helmet use and call for longer-term evaluation.

“Given how far video passthrough headsets have come, it is time to devote serious academic thought to the psychological and behavioral effects of this technology,” said Jeremy Bailenson, Thomas More Storke Professor at the Stanford School of Humanities and Sciences and founding director of the Virtual Human Interaction Lab (VHIL). “We want to understand the implications of living a life in which we rely on relaying for hours each day to see the world around us.”

Advantages of relay

For the study, 10 VHIL researchers and Bailenson himself spent at least 140 minutes over two or three sessions wearing Meta Quest 3 passthrough video headsets, which became widely available in October 2023.

The researchers engaged in a wide range of activities such as conversing, walking outside, playing games, eating, and cooking food. For safety reasons, given fears of tripping over objects or encountering moving people or vehicles, an attendant not wearing a headset remained present at all times.

Study participants attempted to examine the experience from both a practical and subjective perspective as well as a distant clinical perspective. “We took an observational approach, closer to that of naturalists, and really dove into this medium in an exploratory way,” said James Brown, co-author of the study and a master’s student in the Symbolic Systems program. .

In general, researchers found that they liked many aspects of filtering reality through the passthrough filter. “For many of us, wearing headphones in public was exciting,” said Monique Tania Santoso, study co-author and doctoral student in the Department of Communication.

“It was a very novel experience to be in these headsets while walking around campus, interacting with strangers and even buying coffee,” said co-author Portia Wang, a second-year master’s student. in the Department of Management Sciences and Engineering which studies computational social sciences. .

As for Bailenson, who has long followed the development of video passthrough and remembers first donning a rudimentary device in the late 1990s, the experience was “mind-blowing” in comparison.

“It’s hard to describe until you try it, but it’s like magic with these new helmets,” Bailenson said. “The immediacy of video, stereo color, and the incredible visuals that can be rendered, including walls or objects disappearing, your eyes and brain mostly can’t tell the difference.”

Still not as real as real

As researchers continued to spend time immersed in the video passthrough, significant imperfections became apparent, which impacted how users felt and would likely cause issues with frequent headset wear .

In the headset, peripheral vision is lost and users can only perceive about half of what humans normally see. And gadgets still can’t match the sharpness of natural vision. Distortion also occurs – a sort of “fun mirror” effect with shapes and dimensions of objects appearing artificial or morphing – and there was a barely noticeable lag in the display changing when users moved head towards a new view.

“Even though the world you’re looking at is real, it certainly has a video game-like ‘otherness’ to it,” Brown said.

These problems manifested themselves in the fact that users often underestimated distances to objects. For example, giving high fives proved difficult, and when users tried to bring a spoon to their mouth while eating, the view from the headset suggested that the spoon had reached their lips, when in reality the spoon hovered a few inches away.

While helmet wearers have learned to account for these inaccuracies, Bailenson’s team is concerned about the extent to which such overcompensation might persist after prolonged helmet use.

“The companies that make these headsets want you to wear them all day, but what are the consequences and how long do they last?” » Bailenson said. “A plausible scenario could be going down a flight of stairs and missing a step, or driving a car and misjudging distances.”

All of these effects contributed to a deep sense of what is referred to in this research as “social absence.” Examples of this include “difficulty discerning distant facial expressions,” noted by Wang, and “lack of gaze,” reported by Santoso. “People in the outside world have become very absent, like we’re watching them on TV,” Bailenson said. “The person walking, cycling, or sitting near you didn’t feel physically real.”

A final problem the team encountered during its field testing was simulator sickness, a type of motion sickness long documented in virtual reality and first-person games.

“When your eyes see the world moving one way and your body feels it differently, simulator sickness can follow,” Bailenson said. “I was surprised because all 11 of us in this study are veterans of headsets, but even after relatively short periods of use, we tended to feel uncomfortable.”

Adaptation and moderation

Given their experiences, the Stanford researchers recommend that users of mixed reality headsets proceed with caution when adapting to the medium rather than diving into day-long binges.

Bailenson specifically recommends that users of mixed reality products, as well as headset manufacturers themselves, consider reducing the time spent in the headset and taking breaks.

“Pass-through video headsets have great potential for all kinds of applications,” Bailenson said. “But there are also pitfalls that can harm the user experience, from feelings of social absence to motion sickness to after-effects that could even be dangerous.”

Bailenson is a professor in the Department of Communication, a principal investigator at the Stanford Woods Institute for the Environment, and a member of Stanford Bio-X, the Wu Tsai Human Performance Alliance, and the Wu Tsai Neurosciences Institute.

Other Stanford authors include Brian Beams, VHIL laboratory director; graduate students Cyan DeVeaux, Eugy Han, Tara Srirangarajan and Yujie Tao; and postdoctoral researcher Anna CM Queiroz. Co-author Rabindra Ratan is from Michigan State University.