Optical invention improves camera capabilities

Butterflies have a broader view of the world than humans, including more colors and the direction of light field oscillation, or polarization. This special ability allows them to navigate precisely, forage for food, and communicate with each other. Other species, such as the mantis shrimp, can see an even broader spectrum of light, as well as the circular polarization, or rotational states, of light waves. They use this ability to transmit a “love code,” which helps them find and be discovered by their mates.

Inspired by these capabilities in the animal kingdom, a team of researchers at Penn State College of Engineering has developed an ultrathin optical element called a metasurface, which can attach to a conventional camera and encode the spectral and polarization data of captured images in a snapshot or video using tiny antenna-like nanostructures that adapt the properties of light. A machine learning framework, also developed by the team, then decodes this multidimensional visual information in real time on a standard laptop.

The researchers published their work in Scientific progress.

“As the animal world shows us, aspects of light beyond what we can see with our eyes contain information that we can use in many applications,” said Xingjie Ni, associate professor of electrical engineering and lead author of the study. “To do this, we transformed a conventional camera into a compact and lightweight hyperspectro-polarimetric camera by integrating our metasurface into it.”

Hyperspectral and polarimetric cameras, which are often bulky and expensive to produce, capture either spectrum data or polarization data, but not both simultaneously, Ni explains. In contrast, when placed between a camera’s lens and sensors, the inexpensive-to-manufacture three-millimeter-by-three-millimeter metasurface captures both types of imaging data simultaneously and immediately transmits them to a computer.

The raw images then need to be decoded to reveal spectral and polarization information. To achieve this, Bofeng Liu, a PhD student in electrical engineering and co-author of the paper, built a machine learning framework trained on 1.8 million images using data augmentation techniques.

“At 28 frames per second, limited mainly by the speed of the camera used, we are able to quickly retrieve spectral and polarization information through our neural network,” Liu said. “This allows us to capture and visualize image data in real time.”

The researchers tested their metasurface and neural network by recording videos of transparent “PSU” letters under different laser lights. They also captured images of the glorious beetle, which is known to reflect circularly polarized light visible to other species of its kind.

Immediate access to hyperspectro-polarimetric information of different objects could benefit consumers if the technology were commercialized, Ni said.

“We could take our camera with us to supermarkets, take pictures and assess the freshness of fruits and vegetables on the shelves before buying them,” Ni says. “This augmented camera opens a window to the invisible world.”

Additionally, in biomedical applications, Ni said, hyperspectropolarimetric information could be used to differentiate between material and structural properties of body tissues, potentially aiding in the diagnosis of cancer cells.

This work builds on Ni’s previous research and development of other metasurfaces, including one that mimics the processing power of the human eye, and metalenses, including one capable of imaging distant objects, including the moon.

In addition to Ni and Liu, co-authors include co-corresponding author Zhiwen Liu, professor of electrical engineering at Penn State; Hyun-Ju Ahn, postdoctoral researcher in electrical engineering; and electrical engineering graduate students Lidan Zhang, Chen Zhou, Yiming Ding, Shengyuan Chang, Yao Duan, Md. Tarek Rathman, Tunan Xia and Xi Chen.