Integrated photonic encoder for low power and high-speed image processing

Wang, Xiao; Redding, Brandon; Karl, Nicholas; Long, Christopher; Zhu, Zheyuan; Skowronek, James; Pang, Shuo; Brady, David; Sarma, Raktim

doi:10.1038/s41467-024-48099-2

Integrated photonic encoder for low power and high-speed image processing

Journal Article · Mon May 27 00:00:00 EDT 2024 · Nature Communications

DOI:https://doi.org/10.1038/s41467-024-48099-2· OSTI ID:2367233

Wang, Xiao; ; ; Long, Christopher; ; ; Pang, Shuo; Brady, David;

Abstract

Modern lens designs are capable of resolving greater than 10 gigapixels, while advances in camera frame-rate and hyperspectral imaging have made data acquisition rates of Terapixel/second a real possibility. The main bottlenecks preventing such high data-rate systems are power consumption and data storage. In this work, we show that analog photonic encoders could address this challenge, enabling high-speed image compression using orders-of-magnitude lower power than digital electronics. Our approach relies on a silicon-photonics front-end to compress raw image data, foregoing energy-intensive image conditioning and reducing data storage requirements. The compression scheme uses a passive disordered photonic structure to perform kernel-type random projections of the raw image data with minimal power consumption and low latency. A back-end neural network can then reconstruct the original images with structural similarity exceeding 90%. This scheme has the potential to process data streams exceeding Terapixel/second using less than 100 fJ/pixel, providing a path to ultra-high-resolution data and image acquisition systems.

View Journal Article

Sponsoring Organization:: USDOE

OSTI ID:: 2367233

Journal Information:: Nature Communications, Journal Name: Nature Communications Journal Issue: 1 Vol. 15; ISSN 2041-1723

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (50)

Analog Optical Computing for Artificial Intelligence Wu, Jiamin; Lin, Xing; Guo, Yuchen Engineering, Vol. 10 https://doi.org/10.1016/j.eng.2021.06.021	journal	March 2022
Autoencoders Based Deep Learner for Image Denoising Bajaj, Komal; Singh, Dushyant Kumar; Ansari, Mohd. Aquib Procedia Computer Science, Vol. 171 https://doi.org/10.1016/j.procs.2020.04.164	journal	January 2020
Multiscale gigapixel photography Brady, D. J.; Gehm, M. E.; Stack, R. A. Nature, Vol. 486, Issue 7403 https://doi.org/10.1038/nature11150	journal	June 2012
Experimental demonstration of reservoir computing on a silicon photonics chip Vandoorne, Kristof; Mechet, Pauline; Van Vaerenbergh, Thomas Nature Communications, Vol. 5, Issue 1 https://doi.org/10.1038/ncomms4541	journal	March 2014
Compact spectrometer based on a disordered photonic chip Redding, Brandon; Liew, Seng Fatt; Sarma, Raktim Nature Photonics, Vol. 7, Issue 9 https://doi.org/10.1038/nphoton.2013.190	journal	July 2013
Analog optical computing Solli, Daniel R.; Jalali, Bahram Nature Photonics, Vol. 9, Issue 11 https://doi.org/10.1038/nphoton.2015.208	journal	October 2015
Femtofarad optoelectronic integration demonstrating energy-saving signal conversion and nonlinear functions Nozaki, Kengo; Matsuo, Shinji; Fujii, Takuro Nature Photonics, Vol. 13, Issue 7 https://doi.org/10.1038/s41566-019-0397-3	journal	April 2019
Image sensing with multilayer nonlinear optical neural networks Wang, Tianyu; Sohoni, Mandar M.; Wright, Logan G. Nature Photonics, Vol. 17, Issue 5 https://doi.org/10.1038/s41566-023-01170-8	journal	March 2023
An optical-frequency synthesizer using integrated photonics Spencer, Daryl T.; Drake, Tara; Briles, Travis C. Nature, Vol. 557, Issue 7703 https://doi.org/10.1038/s41586-018-0065-7	journal	April 2018
Inference in artificial intelligence with deep optics and photonics Wetzstein, Gordon; Ozcan, Aydogan; Gigan, Sylvain Nature, Vol. 588, Issue 7836 https://doi.org/10.1038/s41586-020-2973-6	journal	December 2020
An on-chip photonic deep neural network for image classification Ashtiani, Farshid; Geers, Alexander J.; Aflatouni, Firooz Nature, Vol. 606, Issue 7914 https://doi.org/10.1038/s41586-022-04714-0	journal	June 2022
All-analog photoelectronic chip for high-speed vision tasks Chen, Yitong; Nazhamaiti, Maimaiti; Xu, Han Nature, Vol. 623, Issue 7985 https://doi.org/10.1038/s41586-023-06558-8	journal	October 2023
Imaging With Nature: Compressive Imaging Using a Multiply Scattering Medium Liutkus, Antoine; Martina, David; Popoff, Sébastien Scientific Reports, Vol. 4, Issue 1 https://doi.org/10.1038/srep05552	journal	July 2014
Thermo-optic coefficient of silicon at 1550 nm and cryogenic temperatures Komma, J.; Schwarz, C.; Hofmann, G. Applied Physics Letters, Vol. 101, Issue 4 https://doi.org/10.1063/1.4738989	journal	July 2012
Achieving beyond-100-GHz large-signal modulation bandwidth in hybrid silicon photonics Mach Zehnder modulators using thin film lithium niobate Wang, Xiaoxi; Weigel, Peter O.; Zhao, Jie APL Photonics, Vol. 4, Issue 9 https://doi.org/10.1063/1.5115243	journal	September 2019
Extensions of Lipschitz mappings into a Hilbert space Johnson, William B.; Lindenstrauss, Joram Conference on Modern Analysis and Probability https://doi.org/10.1090/conm/026/737400	book	January 1984
Position-Dependent Diffusion of Light in Disordered Waveguides Yamilov, Alexey G.; Sarma, Raktim; Redding, Brandon Physical Review Letters, Vol. 112, Issue 2 https://doi.org/10.1103/PhysRevLett.112.023904	journal	January 2014
Control of Energy Density inside a Disordered Medium by Coupling to Open or Closed Channels Sarma, Raktim; Yamilov, Alexey G.; Petrenko, Sasha Physical Review Letters, Vol. 117, Issue 8 https://doi.org/10.1103/PhysRevLett.117.086803	journal	August 2016
A tutorial on modern lossy wavelet image compression: foundations of JPEG 2000 Usevitch, B. E. IEEE Signal Processing Magazine, Vol. 18, Issue 5 https://doi.org/10.1109/79.952803	journal	January 2001
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Enhanced Deep Residual Networks for Single Image Super-Resolution Lim, Bee; Son, Sanghyun; Kim, Heewon 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) https://doi.org/10.1109/CVPRW.2017.151	conference	July 2017
Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics Baek, Seung-Hwan; Ikoma, Hayato; Jeon, Daniel S. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV48922.2021.00265	conference	October 2021
ResUNet++: An Advanced Architecture for Medical Image Segmentation Jha, Debesh; Smedsrud, Pia H.; Riegler, Michael A. 2019 IEEE International Symposium on Multimedia (ISM) https://doi.org/10.1109/ISM46123.2019.00049	conference	December 2019
Review of Silicon Photonics Technology and Platform Development Siew, S. Y.; Li, B.; Gao, F. Journal of Lightwave Technology, Vol. 39, Issue 13 https://doi.org/10.1109/JLT.2021.3066203	journal	July 2021
Road Extraction by Deep Residual U-Net Zhang, Zhengxin; Liu, Qingjie; Wang, Yunhong IEEE Geoscience and Remote Sensing Letters, Vol. 15, Issue 5 https://doi.org/10.1109/LGRS.2018.2802944	journal	May 2018
Compressed sensing Donoho, D. L. IEEE Transactions on Information Theory, Vol. 52, Issue 4 https://doi.org/10.1109/TIT.2006.871582	journal	April 2006
Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? Candes, Emmanuel J.; Tao, Terence IEEE Transactions on Information Theory, Vol. 52, Issue 12, p. 5406-5425 https://doi.org/10.1109/TIT.2006.885507	journal	January 2006
Image Compression Based on Compressive Sensing: End-to-End Comparison With JPEG Yuan, Xin; Haimi-Cohen, Raziel IEEE Transactions on Multimedia, Vol. 22, Issue 11 https://doi.org/10.1109/TMM.2020.2967646	journal	November 2020
Spectrally encoded single-pixel machine vision using diffractive networks Li, Jingxi; Mengu, Deniz; Yardimci, Nezih T. Science Advances, Vol. 7, Issue 13 https://doi.org/10.1126/sciadv.abd7690	journal	March 2021
Compressive Sampling for Array Cameras Yan, Xuefei; Brady, David J.; Zhang, Weiping SIAM Journal on Imaging Sciences, Vol. 14, Issue 1 https://doi.org/10.1137/19M1283914	journal	January 2021
The JPEG still picture compression standard Wallace, Gregory K. Communications of the ACM, Vol. 34, Issue 4 https://doi.org/10.1145/103085.103089	journal	April 1991
Noise analysis of spectrometers based on speckle pattern reconstruction Redding, Brandon; Popoff, Sebastien M.; Bromberg, Yaron Applied Optics, Vol. 53, Issue 3 https://doi.org/10.1364/AO.53.000410	journal	January 2014
Range performance of the DARPA AWARE wide field-of-view visible imager Nichols, J. M.; Judd, K. P.; Olson, C. C. Applied Optics, Vol. 55, Issue 16 https://doi.org/10.1364/AO.55.004478	journal	June 2016
Coherent dimension reduction with integrated photonic circuits exploiting tailored disorder Wendland, Daniel; Becker, Marlon; Brückerhoff-Plückelmann, Frank Journal of the Optical Society of America B, Vol. 40, Issue 3 https://doi.org/10.1364/JOSAB.479898	journal	February 2023
Photonic analog-to-digital converters Valley, George C. Optics Express, Vol. 15, Issue 5 https://doi.org/10.1364/OE.15.001955	journal	January 2007
25Gb/s 1V-driving CMOS ring modulator with integrated thermal tuning Li, Guoliang; Zheng, Xuezhe; Yao, Jin Optics Express, Vol. 19, Issue 21, p. 20435-20443 https://doi.org/10.1364/OE.19.020435	journal	January 2011
Zero-bias 40Gbit/s germanium waveguide photodetector on silicon Vivien, Laurent; Polzer, Andreas; Marris-Morini, Delphine Optics Express, Vol. 20, Issue 2 https://doi.org/10.1364/OE.20.001096	journal	January 2012
Energy consumption in optical modulators for interconnects Miller, David A. B. Optics Express, Vol. 20, Issue S2 https://doi.org/10.1364/OE.20.00A293	journal	January 2012
Silicon photonic physical unclonable function Grubel, Brian C.; Bosworth, Bryan T.; Kossey, Michael R. Optics Express, Vol. 25, Issue 11 https://doi.org/10.1364/OE.25.012710	journal	May 2017
Galilean monocentric multiscale optical systems Pang, Wubin; Brady, David J. Optics Express, Vol. 25, Issue 17 https://doi.org/10.1364/OE.25.020332	journal	August 2017
Photonic integrated circuit based compressive sensing radio frequency receiver using waveguide speckle Borlaug, David B.; Estrella, Steven; Boone, Carl T. Optics Express, Vol. 29, Issue 13 https://doi.org/10.1364/OE.408565	journal	June 2021
High wall-plug efficiency and narrow linewidth III-V-on-silicon C-band DFB laser diodes Vaskasi, Javad Rahimi; Singh, Nishant; Van Kerrebrouck, Joris Optics Express, Vol. 30, Issue 15 https://doi.org/10.1364/OE.462051	journal	July 2022
Evanescently coupled multimode spiral spectrometer Redding, Brandon; Fatt Liew, Seng; Bromberg, Yaron Optica, Vol. 3, Issue 9 https://doi.org/10.1364/OPTICA.3.000956	journal	January 2016
Parallel cameras Brady, David J.; Pang, Wubin; Li, Han Optica, Vol. 5, Issue 2 https://doi.org/10.1364/OPTICA.5.000127	journal	January 2018
Modulators in Silicon Photonics—Heterogenous Integration & and Beyond Mulcahy, Jack; Peters, Frank H.; Dai, Xing Photonics, Vol. 9, Issue 1 https://doi.org/10.3390/photonics9010040	journal	January 2022
Adam: A Method for Stochastic Optimization Kingma, Diederik P.; Ba, Jimmy arXiv https://doi.org/10.48550/arXiv.1412.6980	preprint	January 2014
Lossy Image Compression with Compressive Autoencoders Theis, Lucas; Shi, Wenzhe; Cunningham, Andrew arXiv https://doi.org/10.48550/arXiv.1703.00395	preprint	January 2017
Enhanced Deep Residual Networks for Single Image Super-Resolution Lim, Bee; Son, Sanghyun; Kim, Heewon arXiv https://doi.org/10.48550/arXiv.1707.02921	preprint	January 2017
Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters Havasi, Marton; Peharz, Robert; Hernández-Lobato, José Miguel arXiv https://doi.org/10.48550/arXiv.1810.00440	preprint	January 2018
Autoencoders Bank, Dor; Koenigstein, Noam; Giryes, Raja arXiv https://doi.org/10.48550/arXiv.2003.05991	preprint	January 2020

Similar Records

Zero-Power Analog Optical Processing

Technical Report · Sun Sep 01 00:00:00 EDT 2024 · OSTI ID:2462898

Real-Time Digital Signal Processing for a Fourier Transform Hyperspectral Imager

Technical Report · Tue Jun 01 00:00:00 EDT 1999 · OSTI ID:8363

Integrated photonic encoder for low power and high-speed image processing

Citation Formats

References (50)

Similar Records

Related Subjects