Multichannel meta-imagers for accelerating machine vision

Zheng, Hanyu; Liu, Quan; Kravchenko, Ivan I.; Zhang, Xiaomeng; Huo, Yuankai; Valentine, Jason G.

doi:10.1038/s41565-023-01557-2

Title: Multichannel meta-imagers for accelerating machine vision

Journal Article · 03 January 2024 · Nature Nanotechnology

DOI: https://doi.org/10.1038/s41565-023-01557-2 · OSTI ID:2345321

^[1]; Liu, Quan ^[1];

^[2]; Zhang, Xiaomeng ^[1]; Huo, Yuankai ^[1];

^[1]

Vanderbilt Univ., Nashville, TN (United States)
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Center for Nanophase Materials Sciences (CNMS)

Rapid developments in machine vision technology have impacted a variety of applications, such as medical devices and autonomous driving systems. These achievements, however, typically necessitate digital neural networks with the downside of heavy computational requirements and consequent high energy consumption. As a result, real-time decision-making is hindered when computational resources are not readily accessible. Here we report a meta-imager designed to work together with a digital back end to offload computationally expensive convolution operations into high-speed, low-power optics. Further, in this architecture, metasurfaces enable both angle and polarization multiplexing to create multiple information channels that perform positively and negatively valued convolution operations in a single shot. We use our meta-imager for object classification, achieving 98.6% accuracy in handwritten digits and 88.8% accuracy in fashion images. Owing to its compactness, high speed and low power consumption, our approach could find a wide range of applications in artificial intelligence and machine vision applications.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Center for Nanophase Materials Sciences (CNMS)

Sponsoring Organization:: Defense Advanced Research Projects Agency (DARPA); US Department of the Navy, Office of Naval Research (ONR); USDOE Office of Science (SC), Basic Energy Sciences (BES)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 2345321

Journal Information:: Nature Nanotechnology, Journal Name: Nature Nanotechnology Journal Issue: 4 Vol. 19; ISSN 1748-3387

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United States

Language:: English

References (51)

Incoherent Optoelectronic Differentiation Based on Optimized Multilayer Films Zhang, Xiaomeng; Bai, Benfeng; Sun, Hong‐Bo Laser & Photonics Reviews, Vol. 16, Issue 9 https://doi.org/10.1002/lpor.202200038	journal	July 2022
Scaling equations for the accurate prediction of CMOS device performance from 180 nm to 7 nm Stillmaker, Aaron; Baas, Bevan Integration, Vol. 58 https://doi.org/10.1016/j.vlsi.2017.02.002	journal	June 2017
High-Index Dielectric Metasurfaces Performing Mathematical Operations Cordaro, Andrea; Kwon, Hoyeong; Sounas, Dimitrios Nano Letters, Vol. 19, Issue 12 https://doi.org/10.1021/acs.nanolett.9b02477	journal	October 2019
Compound Meta-Optics for Complete and Loss-Less Field Control Zheng, Hanyu; He, Mingze; Zhou, You ACS Nano, Vol. 16, Issue 9 https://doi.org/10.1021/acsnano.2c06248	journal	August 2022
Angular Transmission Response of In-Plane Symmetry-Breaking Quasi-BIC All-Dielectric Metasurfaces Levanon, Nir; Indukuri, S. R. K. Chaitanya; Frydendahl, Christian ACS Photonics, Vol. 9, Issue 11 https://doi.org/10.1021/acsphotonics.2c01069	journal	November 2022
Compact Incoherent Image Differentiation with Nanophotonic Structures Wang, Haiwen; Guo, Cheng; Zhao, Zhexin ACS Photonics, Vol. 7, Issue 2 https://doi.org/10.1021/acsphotonics.9b01465	journal	January 2020
Deep learning LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey Nature, Vol. 521, Issue 7553 https://doi.org/10.1038/nature14539	journal	May 2015
Multichannel vectorial holographic display and encryption Zhao, Ruizhe; Sain, Basudeb; Wei, Qunshuo Light: Science & Applications, Vol. 7, Issue 1 https://doi.org/10.1038/s41377-018-0091-0	journal	November 2018
Intelligent metasurface imager and recognizer Li, Lianlin; Shuang, Ya; Ma, Qian Light: Science & Applications, Vol. 8, Issue 1 https://doi.org/10.1038/s41377-019-0209-z	journal	October 2019
Performing optical logic operations by a diffractive neural network Qian, Chao; Lin, Xiao; Lin, Xiaobin Light: Science & Applications, Vol. 9, Issue 1 https://doi.org/10.1038/s41377-020-0303-2	journal	April 2020
Ultracompact meta-imagers for arbitrary all-optical convolution Fu, Weiwei; Zhao, Dong; Li, Ziqin Light: Science & Applications, Vol. 11, Issue 1 https://doi.org/10.1038/s41377-022-00752-5	journal	March 2022
LOEN: Lensless opto-electronic neural network empowered machine vision Shi, Wanxin; Huang, Zheng; Huang, Honghao Light: Science & Applications, Vol. 11, Issue 1 https://doi.org/10.1038/s41377-022-00809-5	journal	May 2022
Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible Luo, Xuhao; Hu, Yueqiang; Ou, Xiangnian Light: Science & Applications, Vol. 11, Issue 1 https://doi.org/10.1038/s41377-022-00844-2	journal	May 2022
Programmable phase-change metasurfaces on waveguides for multimode photonic convolutional neural network Wu, Changming; Yu, Heshan; Lee, Seokhyeong Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-020-20365-z	journal	January 2021
An optical neural chip for implementing complex-valued neural network Zhang, H.; Gu, M.; Jiang, X. D. Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-020-20719-7	journal	January 2021
Pixelated bifunctional metasurface-driven dynamic vectorial holographic color prints for photonic security platform Kim, Inki; Jang, Jaehyuck; Kim, Gyeongtae Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-021-23814-5	journal	June 2021
Photonic machine learning with on-chip diffractive optics Fu, Tingzhao; Zang, Yubin; Huang, Yuyao Nature Communications, Vol. 14, Issue 1 https://doi.org/10.1038/s41467-022-35772-7	journal	January 2023
Monocular metasurface camera for passive single-shot 4D imaging Shen, Zicheng; Zhao, Feng; Jin, Chunqi Nature Communications, Vol. 14, Issue 1 https://doi.org/10.1038/s41467-023-36812-6	journal	February 2023
Scalable manufacturing of high-index atomic layer–polymer hybrid metasurfaces for metaphotonics in the visible Kim, Joohoon; Seong, Junhwa; Kim, Wonjoong Nature Materials, Vol. 22, Issue 4 https://doi.org/10.1038/s41563-023-01485-5	journal	March 2023
Single-shot quantitative phase gradient microscopy using a system of multifunctional metasurfaces Kwon, Hyounghan; Arbabi, Ehsan; Kamali, Seyedeh Mahsa Nature Photonics, Vol. 14, Issue 2 https://doi.org/10.1038/s41566-019-0536-x	journal	October 2019
Photonics for artificial intelligence and neuromorphic computing Shastri, Bhavin J.; Tait, Alexander N.; Ferreira de Lima, T. Nature Photonics, Vol. 15, Issue 2 https://doi.org/10.1038/s41566-020-00754-y	journal	January 2021
Flat optics for image differentiation Zhou, You; Zheng, Hanyu; Kravchenko, Ivan I. Nature Photonics, Vol. 14, Issue 5 https://doi.org/10.1038/s41566-020-0591-3	journal	February 2020
Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit Zhou, Tiankuang; Lin, Xing; Wu, Jiamin Nature Photonics, Vol. 15, Issue 5 https://doi.org/10.1038/s41566-021-00796-w	journal	April 2021
Superconducting detectors count more photons Bartley, Tim J. Nature Photonics, Vol. 17, Issue 1 https://doi.org/10.1038/s41566-022-01128-2	journal	December 2022
11 TOPS photonic convolutional accelerator for optical neural networks Xu, Xingyuan; Tan, Mengxi; Corcoran, Bill Nature, Vol. 589, Issue 7840 https://doi.org/10.1038/s41586-020-03063-0	journal	January 2021
Parallel convolutional processing using an integrated photonic tensor core Feldmann, J.; Youngblood, N.; Karpov, M. Nature, Vol. 589, Issue 7840 https://doi.org/10.1038/s41586-020-03070-1	journal	January 2021
Ultrafast machine vision with 2D material neural network image sensors Mennel, Lukas; Symonowicz, Joanna; Wachter, Stefan Nature, Vol. 579, Issue 7797 https://doi.org/10.1038/s41586-020-2038-x	journal	March 2020
Inference in artificial intelligence with deep optics and photonics Wetzstein, Gordon; Ozcan, Aydogan; Gigan, Sylvain Nature, Vol. 588, Issue 7836 https://doi.org/10.1038/s41586-020-2973-6	journal	December 2020
An on-chip photonic deep neural network for image classification Ashtiani, Farshid; Geers, Alexander J.; Aflatouni, Firooz Nature, Vol. 606, Issue 7914 https://doi.org/10.1038/s41586-022-04714-0	journal	June 2022
Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification Chang, Julie; Sitzmann, Vincent; Dun, Xiong Scientific Reports, Vol. 8, Issue 1 https://doi.org/10.1038/s41598-018-30619-y	journal	August 2018
High-NA optical edge detection via optimized multilayer films Xue, Wenjin; Miller, Owen D. Journal of Optics, Vol. 23, Issue 12 https://doi.org/10.1088/2040-8986/ac370c	journal	November 2021
Gradient-based learning applied to document recognition Lecun, Y.; Bottou, L.; Bengio, Y. Proceedings of the IEEE, Vol. 86, Issue 11 https://doi.org/10.1109/5.726791	journal	January 1998
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
A General and Adaptive Robust Loss Function Barron, Jonathan T. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00446	conference	June 2019
RepVGG: Making VGG-style ConvNets Great Again Ding, Xiaohan; Zhang, Xiangyu; Ma, Ningning 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR46437.2021.01352	conference	June 2021
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs Ding, Xiaohan; Zhang, Xiangyu; Han, Jungong 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.01166	conference	June 2022
Computing Systems for Autonomous Driving: State of the Art and Challenges Liu, Liangkai; Lu, Sidi; Zhong, Ren IEEE Internet of Things Journal, Vol. 8, Issue 8 https://doi.org/10.1109/JIOT.2020.3043716	journal	April 2021
Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks Chen, Yu-Hsin; Krishna, Tushar; Emer, Joel S. IEEE Journal of Solid-State Circuits, Vol. 52, Issue 1 https://doi.org/10.1109/JSSC.2016.2616357	journal	January 2017
Fundamental Technologies in Modern Speech Recognition [From the Guest Editors] Furui, Sadaoki; Deng, Li; Gales, Mark IEEE Signal Processing Magazine, Vol. 29, Issue 6 https://doi.org/10.1109/MSP.2012.2209906	journal	November 2012
Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning Wang, Guotai; Li, Wenqi; Zuluaga, Maria A. IEEE Transactions on Medical Imaging, Vol. 37, Issue 7 https://doi.org/10.1109/TMI.2018.2791721	journal	July 2018
Computation at the speed of light: metamaterials for all-optical calculations and neural networks Badloe, Trevon; Lee, Seokho; Rho, Junsuk Advanced Photonics, Vol. 4, Issue 06 https://doi.org/10.1117/1.AP.4.6.064002	journal	December 2022
Snapshot spectral imaging with parallel metasystems McClung, Andrew; Samudrala, Sarath; Torfeh, Mahsa Science Advances, Vol. 6, Issue 38 https://doi.org/10.1126/sciadv.abc7646	journal	September 2020
Meta-optic accelerators for object classifiers Zheng, Hanyu; Liu, Quan; Zhou, You Science Advances, Vol. 8, Issue 30 https://doi.org/10.1126/sciadv.abo6410	journal	July 2022
Single-shot optical neural network Bernstein, Liane; Sludds, Alexander; Panuski, Christopher Science Advances, Vol. 9, Issue 25 https://doi.org/10.1126/sciadv.adg7904	journal	June 2023
Metalenses at visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging Khorasaninejad, Mohammadreza; Chen, Wei Ting; Devlin, Robert C. Science, Vol. 352, Issue 6290 https://doi.org/10.1126/science.aaf6644	journal	June 2016
All-optical machine learning using diffractive deep neural networks Lin, Xing; Rivenson, Yair; Yardimci, Nezih T. Science, Vol. 361, Issue 6406 https://doi.org/10.1126/science.aat8084	journal	July 2018
Metalens-array–based high-dimensional and multiphoton quantum source Li, Lin; Liu, Zexuan; Ren, Xifeng Science, Vol. 368, Issue 6498 https://doi.org/10.1126/science.aba9779	journal	June 2020
Breaking the limitation of polarization multiplexing in optical metasurfaces with engineered noise Xiong, Bo; Liu, Yu; Xu, Yihao Science, Vol. 379, Issue 6629 https://doi.org/10.1126/science.ade5140	journal	January 2023
Icnn Neshatpour, Katayoun; Homayoun, Houman; Sasan, Avesta ACM Transactions on Embedded Computing Systems, Vol. 18, Issue 6 https://doi.org/10.1145/3355553	journal	November 2019
Optical frontend for a convolutional neural network Colburn, Shane; Chu, Yi; Shilzerman, Eli Applied Optics, Vol. 58, Issue 12 https://doi.org/10.1364/AO.58.003179	journal	January 2019
Fast and accurate recurrent neural network acoustic models for speech recognition Sak, Haşim; Senior, Andrew; Rao, Kanishka Interspeech 2015 https://doi.org/10.21437/Interspeech.2015-350	conference	September 2015

Similar Records

Meta-optic accelerators for object classifiers

Journal Article · 2022 · Science Advances · OSTI ID:1883811

Zheng, Hanyu; Liu, Quan; Zhou, You; +3 more

An Efficient Holographic Huygens? Metasurface based on Dielectric Resonant Meta-Atoms.

Journal Article · 2015 · OSTI ID:1427192

Brener, Igal; Chong, Katie; Wang, Lei; +8 more

Quality Control by Artificial Vision

Journal Article · 2009 · Journal of Electronic Imaging · OSTI ID:1036586

Lam, Edmond Y; Gleason, Shaun Scott; Niel, Kurt S

Related Subjects

47 OTHER INSTRUMENTATION

Title: Multichannel meta-imagers for accelerating machine vision

Citation Formats

References (51)

Similar Records

Related Subjects