Snapshot multispectral imaging using a diffractive optical network

Mengu, Deniz; Tabassum, Anika; Jarrahi, Mona; Ozcan, Aydogan

doi:10.1038/s41377-023-01135-0

Title: Snapshot multispectral imaging using a diffractive optical network

Journal Article · 05 April 2023 · Light, Science & Applications

DOI: https://doi.org/10.1038/s41377-023-01135-0 · OSTI ID:1968626

; Tabassum, Anika;

;

Abstract Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal-plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72 λ _m , where λ _m is the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially repeating virtual spectral filter array with 2 × 2 = 4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.

View Journal Article

Cite

Export

Save

Research Organization:: Univ. of California, Los Angeles, CA (United States); University of California, Los Angeles, CA (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Basic Energy Sciences (BES); USDOE Office of Science (SC), Basic Energy Sciences (BES). Materials Sciences & Engineering Division

Grant/Contract Number:: SC0023088

OSTI ID:: 1968626

Journal Information:: Light, Science & Applications, Journal Name: Light, Science & Applications Journal Issue: 1 Vol. 12; ISSN 2047-7538

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (63)

Periodic Dielectric Metasurfaces with High-Efficiency, Multiwavelength Functionalities Sell, David; Yang, Jianji; Doshay, Sage Advanced Optical Materials, Vol. 5, Issue 23 https://doi.org/10.1002/adom.201700645	journal	October 2017
All‐Optical Phase Recovery: Diffractive Computing for Quantitative Phase Imaging Mengu, Deniz; Ozcan, Aydogan Advanced Optical Materials, Vol. 10, Issue 15 https://doi.org/10.1002/adom.202200281	journal	May 2022
Perfect RGB‐IR Color Routers for Sub‐Wavelength Size CMOS Image Sensor Pixels Zhao, Nathan; Catrysse, Peter B.; Fan, Shanhui Advanced Photonics Research, Vol. 2, Issue 3 https://doi.org/10.1002/adpr.202000048	journal	January 2021
Multispectral imaging in biology and medicine: Slices of life Levenson, Richard M.; Mansfield, James R. Cytometry Part A, Vol. 69A, Issue 8 https://doi.org/10.1002/cyto.a.20319	journal	January 2006
Calibration of Space-Multispectral Imaging Sensors Dinguirard, Magdeleine; Slater, Philip N. Remote Sensing of Environment, Vol. 68, Issue 3 https://doi.org/10.1016/S0034-4257(98)00111-4	journal	June 1999
Reflection-optical multispectral imaging method for objective determination of burn depth Eisenbeiß, Werner; Marotz, Jörg; Schrade, Jens-Peter Burns, Vol. 25, Issue 8 https://doi.org/10.1016/S0305-4179(99)00078-9	journal	December 1999
Hyperspectral imaging and multispectral imaging as the novel techniques for detecting defects in raw and processed meat products: Current state-of-the-art research advances Feng, Chao-Hui; Makino, Yoshio; Oshita, Seiichi Food Control, Vol. 84 https://doi.org/10.1016/j.foodcont.2017.07.013	journal	February 2018
Hyperspectral and multispectral imaging for evaluating food safety and quality Qin, Jianwei; Chao, Kuanglin; Kim, Moon S. Journal of Food Engineering, Vol. 118, Issue 2 https://doi.org/10.1016/j.jfoodeng.2013.04.001	journal	September 2013
Multispectral imaging of hematopoietic cells: Where flow meets morphology McGrath, Kathleen E.; Bushnell, Timothy P.; Palis, James Journal of Immunological Methods, Vol. 336, Issue 2 https://doi.org/10.1016/j.jim.2008.04.012	journal	July 2008
Multispectral imaging contributions to global land ice measurements from space Kargel, Jeffrey S.; Abrams, Michael J.; Bishop, Michael P. Remote Sensing of Environment, Vol. 99, Issue 1-2 https://doi.org/10.1016/j.rse.2005.07.004	journal	November 2005
Multispectral imaging – a new tool in seed quality assessment? Boelt, Birte; Shrestha, Santosh; Salimi, Zahra Seed Science Research, Vol. 28, Issue 3 https://doi.org/10.1017/S0960258518000235	journal	June 2018
GaN Metalens for Pixel-Level Full-Color Routing at Visible Light Chen, Bo Han; Wu, Pin Chieh; Su, Vin-Cent Nano Letters, Vol. 17, Issue 10 https://doi.org/10.1021/acs.nanolett.7b03135	journal	September 2017
Single-Layer Bayer Metasurface via Inverse Design Li, Jiahao; Zhang, Qiang; Yang, Hui ACS Photonics, Vol. 9, Issue 8 https://doi.org/10.1021/acsphotonics.2c00990	journal	August 2022
High-Sensitivity Color Imaging Using Pixel-Scale Color Splitters Based on Dielectric Metasurfaces Miyata, Masashi; Nakajima, Mitsumasa; Hashimoto, Toshikazu ACS Photonics, Vol. 6, Issue 6 https://doi.org/10.1021/acsphotonics.9b00042	journal	March 2019
A bimetallic nanoantenna for directional colour routing Shegai, Timur; Chen, Si; Miljković, Vladimir D. Nature Communications, Vol. 2, Issue 1 https://doi.org/10.1038/ncomms1490	journal	September 2011
Efficient colour splitters for high-pixel-density image sensors Nishiwaki, Seiji; Nakamura, Tatsuya; Hiramoto, Masao Nature Photonics, Vol. 7, Issue 3 https://doi.org/10.1038/nphoton.2012.345	journal	February 2013
All-optical information-processing capacity of diffractive surfaces Kulce, Onur; Mengu, Deniz; Rivenson, Yair Light: Science & Applications, Vol. 10, Issue 1 https://doi.org/10.1038/s41377-020-00439-9	journal	January 2021
Nanoprinted high-neuron-density optical linear perceptrons performing near-infrared inference on a CMOS chip Goi, Elena; Chen, Xi; Zhang, Qiming Light: Science & Applications, Vol. 10, Issue 1 https://doi.org/10.1038/s41377-021-00483-z	journal	March 2021
All-optical synthesis of an arbitrary linear transformation using diffractive surfaces Kulce, Onur; Mengu, Deniz; Rivenson, Yair Light: Science & Applications, Vol. 10, Issue 1 https://doi.org/10.1038/s41377-021-00623-5	journal	September 2021
Polarization multiplexed diffractive computing: all-optical implementation of a group of linear transformations through a polarization-encoded diffractive network Li, Jingxi; Hung, Yi-Chun; Kulce, Onur Light: Science & Applications, Vol. 11, Issue 1 https://doi.org/10.1038/s41377-022-00849-x	journal	May 2022
Terahertz pulse shaping using diffractive surfaces Veli, Muhammed; Mengu, Deniz; Yardimci, Nezih T. Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-020-20268-z	journal	January 2021
Pixel-level Bayer-type colour router based on metasurfaces Zou, Xiujuan; Zhang, Youming; Lin, Ruoyu Nature Communications, Vol. 13, Issue 1 https://doi.org/10.1038/s41467-022-31019-7	journal	June 2022
Inference in artificial intelligence with deep optics and photonics Wetzstein, Gordon; Ozcan, Aydogan; Gigan, Sylvain Nature, Vol. 588, Issue 7836 https://doi.org/10.1038/s41586-020-2973-6	journal	December 2020
Machine learning and computation-enabled intelligent sensor design Ballard, Zachary; Brown, Calvin; Madni, Asad M. Nature Machine Intelligence, Vol. 3, Issue 7 https://doi.org/10.1038/s42256-021-00360-9	journal	June 2021
Encoded diffractive optics for full-spectrum computational imaging Heide, Felix; Fu, Qiang; Peng, Yifan Scientific Reports, Vol. 6, Issue 1 https://doi.org/10.1038/srep33543	journal	September 2016
Full-color nanorouter for high-resolution imaging Chen, Mingjie; Wen, Long; Pan, Dahui Nanoscale, Vol. 13, Issue 30 https://doi.org/10.1039/D1NR02166D	journal	January 2021
3D: The next generation near-infrared imaging spectrometer Weitzel, L.; Krabbe, A.; Kroker, H. Astronomy and Astrophysics Supplement Series, Vol. 119, Issue 3, p. 531-546 https://doi.org/10.1051/aas:1996266	journal	November 1996
Real-time imaging with a hyperspectral fovea Fletcher-Holmes, David William; Harvey, Andrew Robert Journal of Optics A: Pure and Applied Optics, Vol. 7, Issue 6 https://doi.org/10.1088/1464-4258/7/6/007	journal	May 2005
Low-cost multispectral imaging system for crop monitoring de Oca, A. Montes; Arreola, L.; Flores, A. 2018 International Conference on Unmanned Aircraft Systems (ICUAS) https://doi.org/10.1109/ICUAS.2018.8453426	conference	June 2018
A novel 3D multispectral vision system based on filter wheel cameras Zhang, Chen; Rosenberger, Maik; Breitbarth, Andreas 2016 IEEE International Conference on Imaging Systems and Techniques (IST) https://doi.org/10.1109/IST.2016.7738235	conference	October 2016
Analysis of Diffractive Optical Neural Networks and Their Integration With Electronic Neural Networks Mengu, Deniz; Luo, Yi; Rivenson, Yair IEEE Journal of Selected Topics in Quantum Electronics, Vol. 26, Issue 1 https://doi.org/10.1109/JSTQE.2019.2921376	journal	January 2020
Hypersectral Imaging for Military and Security Applications: Combining Myriad Processing and Sensing Techniques Shimoni, Michal; Haelterman, Rob; Perneel, Christiaan IEEE Geoscience and Remote Sensing Magazine, Vol. 7, Issue 2 https://doi.org/10.1109/MGRS.2019.2902525	journal	June 2019
Multispectral imaging of paintings Pelagotti, Anna; Mastio, Andrea; Rosa, Alessia IEEE Signal Processing Magazine, Vol. 25, Issue 4 https://doi.org/10.1109/MSP.2008.923095	journal	July 2008
Colored Coded Aperture Design by Concentration of Measure in Compressive Spectral Imaging Arguello, Henry; Arce, Gonzalo R. IEEE Transactions on Image Processing, Vol. 23, Issue 4 https://doi.org/10.1109/TIP.2014.2310125	journal	April 2014
Massively parallel universal linear transformations using a wavelength-multiplexed diffractive optical network Li, Jingxi; Gan, Tianyi; Bai, Bijie Advanced Photonics, Vol. 5, Issue 01 https://doi.org/10.1117/1.AP.5.1.016003	journal	January 2023
Spectrally encoded single-pixel machine vision using diffractive networks Li, Jingxi; Mengu, Deniz; Yardimci, Nezih T. Science Advances, Vol. 7, Issue 13 https://doi.org/10.1126/sciadv.abd7690	journal	March 2021
Pancam Multispectral Imaging Results from the Opportunity Rover at Meridiani Planum Bell, J. F.; Squyres, S. W.; Arvidson, R. E. Science, Vol. 306, Issue 5702 https://doi.org/10.1126/science.1105245	journal	December 2004
All-optical machine learning using diffractive deep neural networks Lin, Xing; Rivenson, Yair; Yardimci, Nezih T. Science, Vol. 361, Issue 6406 https://doi.org/10.1126/science.aat8084	journal	July 2018
Analysis of multispectral imaging with the AstroPath platform informs efficacy of PD-1 blockade Berry, Sneha; Giraldo, Nicolas A.; Green, Benjamin F. Science, Vol. 372, Issue 6547 https://doi.org/10.1126/science.aba2609	journal	June 2021
Spatial-spectral encoded compressive hyperspectral imaging Lin, Xing; Liu, Yebin; Wu, Jiamin ACM Transactions on Graphics, Vol. 33, Issue 6 https://doi.org/10.1145/2661229.2661262	journal	November 2014
Compact snapshot hyperspectral imaging with diffracted rotation Jeon, Daniel S.; Baek, Seung-Hwan; Yi, Shinyoung ACM Transactions on Graphics, Vol. 38, Issue 4 https://doi.org/10.1145/3306346.3322946	journal	July 2019
Identification of pigments by multispectral imaging; a flowchart method Cosentino, Antonino Heritage Science, Vol. 2, Issue 1 https://doi.org/10.1186/2050-7445-2-8	journal	January 2014
Computational imaging without a computer: seeing through random diffusers at the speed of light Luo, Yi; Zhao, Yifan; Li, Jingxi eLight, Vol. 2, Issue 1 https://doi.org/10.1186/s43593-022-00012-4	journal	January 2022
To image, or not to image: class-specific diffractive cameras with all-optical erasure of undesired objects Bai, Bijie; Luo, Yi; Gan, Tianyi eLight, Vol. 2, Issue 1 https://doi.org/10.1186/s43593-022-00021-3	journal	August 2022
Single disperser design for coded aperture snapshot spectral imaging Wagadarikar, Ashwin; John, Renu; Willett, Rebecca Applied Optics, Vol. 47, Issue 10, p. B44-B51 https://doi.org/10.1364/AO.47.000B44	journal	January 2008
Multispectral camera and radiative transfer equation used to depict Leonardo's sfumato in Mona Lisa Elias, Mady; Cotte, Pascal Applied Optics, Vol. 47, Issue 12 https://doi.org/10.1364/AO.47.002146	journal	April 2008
Compressive hyperspectral imaging by random separable projections in both the spatial and the spectral domains August, Yitzhak; Vachman, Chaim; Rivenson, Yair Applied Optics, Vol. 52, Issue 10 https://doi.org/10.1364/AO.52.000D46	journal	March 2013
Free-space beam propagation between arbitrarily oriented planes based on full diffraction theory: a fast Fourier transform approach Delen, N.; Hooker, B. Journal of the Optical Society of America A, Vol. 15, Issue 4 https://doi.org/10.1364/JOSAA.15.000857	journal	January 1998
Fast calculation method for optical diffraction on tilted planes by use of the angular spectrum of plane waves Matsushima, Kyoji; Schimmel, Hagen; Wyrowski, Frank Journal of the Optical Society of America A, Vol. 20, Issue 9 https://doi.org/10.1364/JOSAA.20.001755	journal	January 2003
Snapshot colored compressive spectral imager Correa, Claudia V.; Arguello, Henry; Arce, Gonzalo R. Journal of the Optical Society of America A, Vol. 32, Issue 10 https://doi.org/10.1364/JOSAA.32.001754	journal	September 2015
RGB-NIR multispectral camera Chen, Zhenyue; Wang, Xia; Liang, Rongguang Optics Express, Vol. 22, Issue 5 https://doi.org/10.1364/OE.22.004985	journal	February 2014
Multispectral imaging with a confocal microendoscope Rouse, Andrew R.; Gmitro, Arthur F. Optics Letters, Vol. 25, Issue 23 https://doi.org/10.1364/OL.25.001708	journal	December 2000
Development of a digital-micromirror-device-based multishot snapshot spectral imaging system Wu, Yuehao; Mirza, Iftekhar O.; Arce, Gonzalo R. Optics Letters, Vol. 36, Issue 14 https://doi.org/10.1364/OL.36.002692	journal	July 2011
Dual-coded compressive hyperspectral imaging Lin, Xing; Wetzstein, Gordon; Liu, Yebin Optics Letters, Vol. 39, Issue 7 https://doi.org/10.1364/OL.39.002044	journal	March 2014
Ultra-high-sensitivity color imaging via a transparent diffractive-filter array and computational optics Wang, Peng; Menon, Rajesh Optica, Vol. 2, Issue 11 https://doi.org/10.1364/OPTICA.2.000933	journal	January 2015
Multifunctional volumetric meta-optics for color and polarization image sensors Camayd-Muñoz, Philip; Ballew, Conner; Roberts, Gregory Optica, Vol. 7, Issue 4 https://doi.org/10.1364/OPTICA.384228	journal	January 2020
Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging Dun, Xiong; Ikoma, Hayato; Wetzstein, Gordon Optica, Vol. 7, Issue 8 https://doi.org/10.1364/OPTICA.394413	journal	August 2020
Shift-variant color-coded diffractive spectral imaging system Arguello, Henry; Pinilla, Samuel; Peng, Yifan Optica, Vol. 8, Issue 11 https://doi.org/10.1364/OPTICA.439142	journal	November 2021
Misalignment resilient diffractive optical networks Mengu, Deniz; Zhao, Yifan; Yardimci, Nezih T. Nanophotonics, Vol. 9, Issue 13 https://doi.org/10.1515/nanoph-2020-0291	journal	July 2020
Diffractive interconnects: all-optical permutation operation using diffractive networks Mengu, Deniz; Zhao, Yifan; Tabassum, Anika Nanophotonics, Vol. 0, Issue 0 https://doi.org/10.1515/nanoph-2022-0358	journal	September 2022
Multispectral imaging and pathology: seeing and doing more Levenson, Richard M.; Fornari, Alessandro; Loda, Massimo Expert Opinion on Medical Diagnostics, Vol. 2, Issue 9 https://doi.org/10.1517/17530059.2.9.1067	journal	September 2008
In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer Halicek, Martin; Fabelo, Himar; Ortega, Samuel Cancers, Vol. 11, Issue 6 https://doi.org/10.3390/cancers11060756	journal	May 2019
Use of Hyperspectral/Multispectral Imaging in Gastroenterology. Shedding Some–Different–Light into the Dark Ortega, Samuel; Fabelo, Himar; Iakovidis, Dimitris Journal of Clinical Medicine, Vol. 8, Issue 1 https://doi.org/10.3390/jcm8010036	journal	January 2019

Similar Records

Multispectral Quantitative Phase Imaging Using a Diffractive Optical Network

Journal Article · 2023 · Advanced Intelligent Systems · OSTI ID:1996868

Shen, Che-Yung; Li, Jingxi; Mengu, Deniz; +1 more

Multispectral Focal Plane Assembly for Satellite Remote Sensing

Conference · 1997 · OSTI ID:622892

Rienstra, J; Ballard, M

Prototype focal plane assembly for multispectral remote sensing

Technical Report · 1994 · OSTI ID:112935

Rienstra, J L; Vampola, J A

Related Subjects

36 MATERIALS SCIENCE
42 ENGINEERING
97 MATHEMATICS AND COMPUTING
Imaging and sensing
Optical materials and structures

Title: Snapshot multispectral imaging using a diffractive optical network

Citation Formats

References (63)

Similar Records

Related Subjects