DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Imaging through glass diffusers using densely connected convolutional networks

Abstract

Computational imaging through scatter generally is accomplished by first characterizing the scattering medium so that its forward operator is obtained and then imposing additional priors in the form of regularizers on the reconstruction functional to improve the condition of the originally ill-posed inverse problem. In the functional, the forward operator and regularizer must be entered explicitly or parametrically (e.g., scattering matrices and dictionaries, respectively). However, the process of determining these representations is often incomplete, prone to errors, or infeasible. Recently, deep learning architectures have been proposed to instead learn both the forward operator and regularizer through examples. Here, we propose for the first time, to our knowledge, a convolutional neural network architecture called “IDiffNet” for the problem of imaging through diffuse media and demonstrate that IDiffNet has superior generalization capability through extensive tests with well-calibrated diffusers. We also introduce the negative Pearson correlation coefficient (NPCC) loss function for neural net training and show that the NPCC is more appropriate for spatially sparse objects and strong scattering conditions. Our results show that the convolutional architecture is robust to the choice of prior, as demonstrated by the use of multiple training and testing object databases, and capable of achieving higher space–bandwidth productmore » reconstructions than previously reported.« less

Authors:
; ; ; ;
Publication Date:
Research Org.:
Krell Inst., Inc., Ames, IA (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1459049
Alternate Identifier(s):
OSTI ID: 1502468
Grant/Contract Number:  
FG02-97ER25308
Resource Type:
Published Article
Journal Name:
Optica
Additional Journal Information:
Journal Name: Optica Journal Volume: 5 Journal Issue: 7; Journal ID: ISSN 2334-2536
Publisher:
Optical Society of America
Country of Publication:
United States
Language:
English
Subject:
47 OTHER INSTRUMENTATION

Citation Formats

Li, Shuai, Deng, Mo, Lee, Justin, Sinha, Ayan, and Barbastathis, George. Imaging through glass diffusers using densely connected convolutional networks. United States: N. p., 2018. Web. doi:10.1364/OPTICA.5.000803.
Li, Shuai, Deng, Mo, Lee, Justin, Sinha, Ayan, & Barbastathis, George. Imaging through glass diffusers using densely connected convolutional networks. United States. https://doi.org/10.1364/OPTICA.5.000803
Li, Shuai, Deng, Mo, Lee, Justin, Sinha, Ayan, and Barbastathis, George. Fri . "Imaging through glass diffusers using densely connected convolutional networks". United States. https://doi.org/10.1364/OPTICA.5.000803.
@article{osti_1459049,
title = {Imaging through glass diffusers using densely connected convolutional networks},
author = {Li, Shuai and Deng, Mo and Lee, Justin and Sinha, Ayan and Barbastathis, George},
abstractNote = {Computational imaging through scatter generally is accomplished by first characterizing the scattering medium so that its forward operator is obtained and then imposing additional priors in the form of regularizers on the reconstruction functional to improve the condition of the originally ill-posed inverse problem. In the functional, the forward operator and regularizer must be entered explicitly or parametrically (e.g., scattering matrices and dictionaries, respectively). However, the process of determining these representations is often incomplete, prone to errors, or infeasible. Recently, deep learning architectures have been proposed to instead learn both the forward operator and regularizer through examples. Here, we propose for the first time, to our knowledge, a convolutional neural network architecture called “IDiffNet” for the problem of imaging through diffuse media and demonstrate that IDiffNet has superior generalization capability through extensive tests with well-calibrated diffusers. We also introduce the negative Pearson correlation coefficient (NPCC) loss function for neural net training and show that the NPCC is more appropriate for spatially sparse objects and strong scattering conditions. Our results show that the convolutional architecture is robust to the choice of prior, as demonstrated by the use of multiple training and testing object databases, and capable of achieving higher space–bandwidth product reconstructions than previously reported.},
doi = {10.1364/OPTICA.5.000803},
journal = {Optica},
number = 7,
volume = 5,
place = {United States},
year = {2018},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1364/OPTICA.5.000803

Citation Metrics:
Cited by: 30 works
Citation information provided by
Web of Science

Figures / Tables:

Fig. 1. Fig. 1. : Optical configuration. (a) Experimental arrangement. SF, spatial filter; CL, collimating lens; M, mirror; POL, linear polarizer; BS, beam splitter; SLM, spatial light modulator. (b) Detail of the telescopic imaging system.

Save / Share:

Works referenced in this record:

Widefield lensless imaging through a fiber bundle via speckle correlations
journal, January 2016

  • Porat, Amir; Andresen, Esben Ravn; Rigneault, Hervé
  • Optics Express, Vol. 24, Issue 15
  • DOI: 10.1364/OE.24.016835

Hierarchical classification of images by sparse approximation
journal, December 2013


Object classification through scattering media with deep learning on time resolved measurement
journal, January 2017

  • Satat, Guy; Tancik, Matthew; Gupta, Otkrist
  • Optics Express, Vol. 25, Issue 15
  • DOI: 10.1364/OE.25.017466

Memory Effects in Propagation of Optical Waves through Disordered Media
journal, November 1988


Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information
journal, February 2006

  • Candes, E.J.; Romberg, J.; Tao, T.
  • IEEE Transactions on Information Theory, Vol. 52, Issue 2, p. 489-509
  • DOI: 10.1109/TIT.2005.862083

Generalized optical memory effect
journal, January 2017

  • Osnabrugge, Gerwin; Horstmeyer, Roarke; Papadopoulos, Ioannis N.
  • Optica, Vol. 4, Issue 8
  • DOI: 10.1364/OPTICA.4.000886

Lensless computational imaging through deep learning
journal, January 2017


Image transmission through an opaque material
journal, September 2010

  • Popoff, Sébastien; Lerosey, Geoffroy; Fink, Mathias
  • Nature Communications, Vol. 1, Issue 1
  • DOI: 10.1038/ncomms1078

Single-shot diffuser-encoded light field imaging
conference, May 2016

  • Antipa, Nicholas; Necula, Sylvia; Ng, Ren
  • 2016 IEEE International Conference on Computational Photography (ICCP)
  • DOI: 10.1109/ICCPHOT.2016.7492880

Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy
journal, January 1994


DiffuserCam: lensless single-exposure 3D imaging
journal, December 2017


Measuring the Transmission Matrix in Optics: An Approach to the Study and Control of Light Propagation in Disordered Media
journal, March 2010


Correlations and Fluctuations of Coherent Wave Transmission through Disordered Media
journal, August 1988


ImageNet Large Scale Visual Recognition Challenge
journal, April 2015

  • Russakovsky, Olga; Deng, Jia; Su, Hao
  • International Journal of Computer Vision, Vol. 115, Issue 3
  • DOI: 10.1007/s11263-015-0816-y

Membrane imaging by second-harmonic generation microscopy
journal, January 2000

  • Moreaux, L.; Sandre, O.; Mertz, J.
  • Journal of the Optical Society of America B, Vol. 17, Issue 10
  • DOI: 10.1364/JOSAB.17.001685

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
journal, December 2017

  • Badrinarayanan, Vijay; Kendall, Alex; Cipolla, Roberto
  • IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 12
  • DOI: 10.1109/TPAMI.2016.2644615

Reference-less measurement of the transmission matrix of a highly scattering material using a DMD and phase retrieval techniques
journal, January 2015

  • Drémeau, Angélique; Liutkus, Antoine; Martina, David
  • Optics Express, Vol. 23, Issue 9
  • DOI: 10.1364/OE.23.011898

Non-invasive imaging through opaque scattering layers
journal, November 2012

  • Bertolotti, Jacopo; van Putten, Elbert G.; Blum, Christian
  • Nature, Vol. 491, Issue 7423
  • DOI: 10.1038/nature11578

Speckle-field digital holographic microscopy
journal, January 2009

  • Park, YongKeun; Choi, Wonshik; Yaqoob, Zahid
  • Optics Express, Vol. 17, Issue 15
  • DOI: 10.1364/OE.17.012285

Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations
journal, August 2014


Reconstruction of an object from the modulus of its Fourier transform
journal, January 1978


Optical sectioning in fluorescence microscopy: OPTICAL SECTIONING IN FLUORESCENCE MICROSCOPY
journal, November 2010


3D imaging in volumetric scattering media using phase-space measurements
journal, January 2015

  • Liu, Hsiou-Yuan; Jonas, Eric; Tian, Lei
  • Optics Express, Vol. 23, Issue 11
  • DOI: 10.1364/OE.23.014461

Deep Convolutional Neural Network for Inverse Problems in Imaging
journal, September 2017

  • Jin, Kyong Hwan; McCann, Michael T.; Froustey, Emmanuel
  • IEEE Transactions on Image Processing, Vol. 26, Issue 9
  • DOI: 10.1109/TIP.2017.2713099

Calibration-free imaging through a multicore fiber using speckle scanning microscopy
journal, January 2016

  • Stasio, Nicolino; Moser, Christophe; Psaltis, Demetri
  • Optics Letters, Vol. 41, Issue 13
  • DOI: 10.1364/OL.41.003078

Compressive Holography
journal, January 2009

  • Brady, David J.; Choi, Kerkil; Marks, Daniel L.
  • Optics Express, Vol. 17, Issue 15
  • DOI: 10.1364/OE.17.013040

Learning-based imaging through scattering media
journal, January 2016

  • Horisaki, Ryoichi; Takagi, Ryosuke; Tanida, Jun
  • Optics Express, Vol. 24, Issue 13
  • DOI: 10.1364/OE.24.013738

Wide-field fluorescence sectioning with hybrid speckle and uniform-illumination microscopy
journal, January 2008

  • Lim, Daryl; Chu, Kengyeh K.; Mertz, Jerome
  • Optics Letters, Vol. 33, Issue 16
  • DOI: 10.1364/OL.33.001819

Two-photon laser scanning fluorescence microscopy
journal, April 1990


Works referencing / citing this record:

ImageNet Large Scale Visual Recognition Challenge
journal, April 2015

  • Russakovsky, Olga; Deng, Jia; Su, Hao
  • International Journal of Computer Vision, Vol. 115, Issue 3
  • DOI: 10.1007/s11263-015-0816-y

Hierarchical classification of images by sparse approximation
journal, December 2013


Non-invasive imaging through opaque scattering layers
journal, November 2012

  • Bertolotti, Jacopo; van Putten, Elbert G.; Blum, Christian
  • Nature, Vol. 491, Issue 7423
  • DOI: 10.1038/nature11578

Image transmission through an opaque material
journal, September 2010

  • Popoff, Sébastien; Lerosey, Geoffroy; Fink, Mathias
  • Nature Communications, Vol. 1, Issue 1
  • DOI: 10.1038/ncomms1078

Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations
journal, August 2014


Measuring the Transmission Matrix in Optics: An Approach to the Study and Control of Light Propagation in Disordered Media
journal, March 2010


Memory Effects in Propagation of Optical Waves through Disordered Media
journal, November 1988


Correlations and Fluctuations of Coherent Wave Transmission through Disordered Media
journal, August 1988


Single-shot diffuser-encoded light field imaging
conference, May 2016

  • Antipa, Nicholas; Necula, Sylvia; Ng, Ren
  • 2016 IEEE International Conference on Computational Photography (ICCP)
  • DOI: 10.1109/iccphot.2016.7492880

Deep Convolutional Neural Network for Inverse Problems in Imaging
journal, September 2017

  • Jin, Kyong Hwan; McCann, Michael T.; Froustey, Emmanuel
  • IEEE Transactions on Image Processing, Vol. 26, Issue 9
  • DOI: 10.1109/tip.2017.2713099

Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information
journal, February 2006

  • Candes, E.J.; Romberg, J.; Tao, T.
  • IEEE Transactions on Information Theory, Vol. 52, Issue 2, p. 489-509
  • DOI: 10.1109/tit.2005.862083

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
journal, December 2017

  • Badrinarayanan, Vijay; Kendall, Alex; Cipolla, Roberto
  • IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 12
  • DOI: 10.1109/tpami.2016.2644615

Optical sectioning in fluorescence microscopy: OPTICAL SECTIONING IN FLUORESCENCE MICROSCOPY
journal, November 2010


Two-photon laser scanning fluorescence microscopy
journal, April 1990


Membrane imaging by second-harmonic generation microscopy
journal, January 2000

  • Moreaux, L.; Sandre, O.; Mertz, J.
  • Journal of the Optical Society of America B, Vol. 17, Issue 10
  • DOI: 10.1364/josab.17.001685

Speckle-field digital holographic microscopy
journal, January 2009

  • Park, YongKeun; Choi, Wonshik; Yaqoob, Zahid
  • Optics Express, Vol. 17, Issue 15
  • DOI: 10.1364/oe.17.012285

Compressive Holography
journal, January 2009

  • Brady, David J.; Choi, Kerkil; Marks, Daniel L.
  • Optics Express, Vol. 17, Issue 15
  • DOI: 10.1364/oe.17.013040

Reference-less measurement of the transmission matrix of a highly scattering material using a DMD and phase retrieval techniques
journal, January 2015

  • Drémeau, Angélique; Liutkus, Antoine; Martina, David
  • Optics Express, Vol. 23, Issue 9
  • DOI: 10.1364/oe.23.011898

3D imaging in volumetric scattering media using phase-space measurements
journal, January 2015

  • Liu, Hsiou-Yuan; Jonas, Eric; Tian, Lei
  • Optics Express, Vol. 23, Issue 11
  • DOI: 10.1364/oe.23.014461

Learning-based imaging through scattering media
journal, January 2016

  • Horisaki, Ryoichi; Takagi, Ryosuke; Tanida, Jun
  • Optics Express, Vol. 24, Issue 13
  • DOI: 10.1364/oe.24.013738

Widefield lensless imaging through a fiber bundle via speckle correlations
journal, January 2016

  • Porat, Amir; Andresen, Esben Ravn; Rigneault, Hervé
  • Optics Express, Vol. 24, Issue 15
  • DOI: 10.1364/oe.24.016835

Object classification through scattering media with deep learning on time resolved measurement
journal, January 2017

  • Satat, Guy; Tancik, Matthew; Gupta, Otkrist
  • Optics Express, Vol. 25, Issue 15
  • DOI: 10.1364/oe.25.017466

Deep learning approach for Fourier ptychography microscopy
journal, January 2018


Learning to decompose the modes in few-mode fibers with deep convolutional neural network
journal, January 2019


One-step robust deep learning phase unwrapping
journal, January 2019


Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy
journal, January 1994


Reconstruction of an object from the modulus of its Fourier transform
journal, January 1978


Wide-field fluorescence sectioning with hybrid speckle and uniform-illumination microscopy
journal, January 2008

  • Lim, Daryl; Chu, Kengyeh K.; Mertz, Jerome
  • Optics Letters, Vol. 33, Issue 16
  • DOI: 10.1364/ol.33.001819

Calibration-free imaging through a multicore fiber using speckle scanning microscopy
journal, January 2016

  • Stasio, Nicolino; Moser, Christophe; Psaltis, Demetri
  • Optics Letters, Vol. 41, Issue 13
  • DOI: 10.1364/ol.41.003078

Generalized optical memory effect
journal, January 2017

  • Osnabrugge, Gerwin; Horstmeyer, Roarke; Papadopoulos, Ioannis N.
  • Optica, Vol. 4, Issue 8
  • DOI: 10.1364/optica.4.000886

Lensless computational imaging through deep learning
journal, January 2017


DiffuserCam: lensless single-exposure 3D imaging
journal, December 2017


Reliable deep-learning-based phase imaging with uncertainty quantification
journal, January 2019


Single-shot multispectral imaging through a thin scatterer
journal, January 2019


Low Photon Count Phase Retrieval Using Deep Learning
journal, December 2018


Low Photon Count Phase Retrieval Using Deep Learning
journal, December 2018


Deep learning approach for Fourier ptychography microscopy
journal, January 2018


Learning to decompose the modes in few-mode fibers with deep convolutional neural network
journal, January 2019


One-step robust deep learning phase unwrapping
journal, January 2019


Reliable deep-learning-based phase imaging with uncertainty quantification
journal, January 2019


Single-shot multispectral imaging through a thin scatterer
journal, January 2019


Deep learning in holography and coherent imaging
journal, September 2019


Deep-learning-based ghost imaging
journal, December 2017


Deep learning-based optical field screening for robust optical diffraction tomography
journal, October 2019


Focusing light through scattering media by reinforced hybrid algorithms
journal, January 2020

  • Luo, Yunqi; Yan, Suxia; Li, Huanhao
  • APL Photonics, Vol. 5, Issue 1
  • DOI: 10.1063/1.5131181

High-resolution limited-angle phase tomography of dense layered objects using deep neural networks
journal, September 2019

  • Goy, Alexandre; Rughoobur, Girish; Li, Shuai
  • Proceedings of the National Academy of Sciences, Vol. 116, Issue 40
  • DOI: 10.1073/pnas.1821378116

Fibre-optic based particle sensing via deep learning
journal, September 2019

  • Grant-Jacob, James A.; Jain, Saurabh; Xie, Yunhui
  • Journal of Physics: Photonics, Vol. 1, Issue 4
  • DOI: 10.1088/2515-7647/ab437b

Superhigh-Resolution Recognition of Optical Vortex Modes Assisted by a Deep-Learning Method
journal, October 2019


Analysis of Phase-Extraction Neural Network (PhENN) performance for lensless quantitative phase imaging
conference, March 2019

  • Li, Shuai; Barbastathis, George; Goy, Alexandre
  • Quantitative Phase Imaging V
  • DOI: 10.1117/12.2513310

The importance of physical pre-processors for quantitative phase retrieval under extremely low photon counts
conference, March 2019

  • Goy, Alexandre; Arthur, Kwabena; Li, Shuai
  • Quantitative Phase Imaging V
  • DOI: 10.1117/12.2513314

Artificial intelligence-assisted light control and computational imaging through scattering media
journal, July 2019

  • Cheng, Shengfu; Li, Huanhao; Luo, Yunqi
  • Journal of Innovative Optical Health Sciences, Vol. 12, Issue 04
  • DOI: 10.1142/s1793545819300064

Brief review on learning-based methods for optical tomography
journal, November 2019

  • Zhang, Lin; Zhang, Guanglei
  • Journal of Innovative Optical Health Sciences, Vol. 12, Issue 06
  • DOI: 10.1142/s1793545819300118

Embedded deep learning in ophthalmology: making ophthalmic imaging smarter
journal, January 2019

  • Teikari, Petteri; Najjar, Raymond P.; Schmetterer, Leopold
  • Therapeutic Advances in Ophthalmology, Vol. 11
  • DOI: 10.1177/2515841419827172

A path to high-quality imaging through disordered optical fibers: a review
journal, January 2019

  • Zhao, Jian; Peysokhan, Mostafa; Antonio-Lopez, Jose Enrique
  • Applied Optics, Vol. 58, Issue 13
  • DOI: 10.1364/ao.58.000d50

Single-shot coherent power-spectrum imaging of objects hidden by opaque scattering media
journal, January 2019

  • Tang, Wusheng; Yang, Jiankun; Yi, Wenjun
  • Applied Optics, Vol. 58, Issue 4
  • DOI: 10.1364/ao.58.001033

DeepLSR: a deep learning approach for laser speckle reduction
journal, January 2019

  • Bobrow, Taylor L.; Mahmood, Faisal; Inserni, Miguel
  • Biomedical Optics Express, Vol. 10, Issue 6
  • DOI: 10.1364/boe.10.002869

High-throughput, volumetric quantitative phase imaging with multiplexed intensity diffraction tomography
journal, January 2019


Inverse scattering for reflection intensity phase microscopy
journal, January 2020

  • Matlock, Alex; Sentenac, Anne; Chaumet, Patrick C.
  • Biomedical Optics Express, Vol. 11, Issue 2
  • DOI: 10.1364/boe.380845

Spectral pre-modulation of training examples enhances the spatial resolution of the phase extraction neural network (PhENN)
journal, January 2018


Light scattering control in transmission and reflection with neural networks
journal, January 2018

  • Turpin, Alex; Vishniakou, Ivan; Seelig, Johannes d.
  • Optics Express, Vol. 26, Issue 23
  • DOI: 10.1364/oe.26.030911

Cycle-consistent deep learning approach to coherent noise reduction in optical diffraction tomography
journal, January 2019


Deep learning the high variability and randomness inside multimode fibers
journal, January 2019


Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging
journal, January 2019


Learned reconstructions for practical mask-based lensless imaging
journal, January 2019

  • Monakhova, Kristina; Yurtsever, Joshua; Kuo, Grace
  • Optics Express, Vol. 27, Issue 20
  • DOI: 10.1364/oe.27.028075

RedCap: residual encoder-decoder capsule network for holographic image reconstruction
journal, January 2020

  • Zeng, Tianjiao; So, Hayden K. -H.; Lam, Edmund Y.
  • Optics Express, Vol. 28, Issue 4
  • DOI: 10.1364/oe.383350

Sub-Nyquist computational ghost imaging with deep learning
journal, January 2020

  • Wu, Heng; Wang, Ruizhou; Zhao, Genping
  • Optics Express, Vol. 28, Issue 3
  • DOI: 10.1364/oe.386976

Seeing through multimode fibers with real-valued intensity transmission matrices
journal, January 2020

  • Zhao, Tianrui; Ourselin, Sebastien; Vercauteren, Tom
  • Optics Express, Vol. 28, Issue 14
  • DOI: 10.1364/oe.396734

Rapid phase retrieval of ultrashort pulses from dispersion scan traces using deep neural networks
journal, January 2019

  • Kleinert, Sven; Tajalli, Ayhan; Nagy, Tamas
  • Optics Letters, Vol. 44, Issue 4
  • DOI: 10.1364/ol.44.000979

Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media
journal, January 2018


On the use of deep learning for computational imaging
journal, January 2019