Real-time machine learning–enhanced hyperspectro-polarimetric imaging via an encoding metasurface

Zhang et al., Sci. Adv. 10, eadp5192 (2024) 4 September 2024 S c i e n c e A d va n c e s | R e s e a r c h A r t i c l e 1 of 10 O P T I C S Real-­time machine learning–enhanced hyperspectro-­polarimetric imaging via an encoding metasurface Lidan Zhang†, Chen Zhou†, Bofeng Liu†, Yimin Ding, Hyun-­Ju Ahn, Shengyuan Chang, Yao Duan, Md Tarek Rahman, Tunan Xia, Xi Chen, Zhiwen Liu*, Xingjie Ni* Light fields carry a wealth of information, including intensity, spectrum, and polarization. However, standard cameras capture only the intensity, disregarding other valuable information. While hyperspectral and polarimetric imaging sys- tems capture spectral and polarization information, respectively, in addition to intensity, they are often bulky, slow, and costly. Here, we have developed an encoding metasurface paired with a neural network enabling a normal camera to acquire hyperspectro-­polarimetric images from a single snapshot. Our experimental results demonstrate that this metasurface-­enhanced camera can accurately resolve full-­Stokes polarization across a broad spectral range (700 to 1150 nanometer) from a single snapshot, achieving a spectral sensitivity as high as 0.23 nanometer. In addition, our sys- tem captures full-­Stokes hyperspectro-­polarimetric video in real time at a rate of 28 frames per second, primarily limited by the camera’s readout rate. Our encoding metasurface offers a compact, fast, and cost-­effective solution for multi- dimensional imaging that effectively uses information within light fields. INTRODUCTION The ability to observe and use multidimensional information of light has been a long-­standing human pursuit, offering a more comprehensive understanding of light-­matter interaction and pro- moting more accurate characterizations across many fields (1–4). For instance, the spectral information can significantly improve the precision of medical diagnosis and agricultural monitoring (5, 6), while the polarization information aids in material classifica- tion and stress analysis (7). Obtaining multidimensional informa- tion within a single system presents substantial advantages. However, current hyperspectral and polarimetric imaging tech- niques often capture these two distinct types of data separately, mak- ing it difficult to rapidly obtain comprehensive information (8–11). Furthermore, conventional spectral and polarization imaging sys- tems, which predominantly rely on diffractive optics or optical filters, tend