skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Lensless computational imaging through deep learning

Abstract

Deep learning has been proven to yield reliably generalizable solutions to numerous classification and decision tasks. Here, we demonstrate for the first time to our knowledge that deep neural networks (DNNs) can be trained to solve end-to-end inverse problems in computational imaging. We experimentally built and tested a lensless imaging system where a DNN was trained to recover phase objects given their propagated intensity diffraction patterns.

Authors:
; ; ;
Publication Date:
Research Org.:
Krell Institute, Ames, IA (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1390388
Alternate Identifier(s):
OSTI ID: 1465165
Grant/Contract Number:  
FG02-97ER25308
Resource Type:
Published Article
Journal Name:
Optica
Additional Journal Information:
Journal Name: Optica Journal Volume: 4 Journal Issue: 9; Journal ID: ISSN 2334-2536
Publisher:
Optical Society of America
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Sinha, Ayan, Lee, Justin, Li, Shuai, and Barbastathis, George. Lensless computational imaging through deep learning. United States: N. p., 2017. Web. https://doi.org/10.1364/OPTICA.4.001117.
Sinha, Ayan, Lee, Justin, Li, Shuai, & Barbastathis, George. Lensless computational imaging through deep learning. United States. https://doi.org/10.1364/OPTICA.4.001117
Sinha, Ayan, Lee, Justin, Li, Shuai, and Barbastathis, George. Fri . "Lensless computational imaging through deep learning". United States. https://doi.org/10.1364/OPTICA.4.001117.
@article{osti_1390388,
title = {Lensless computational imaging through deep learning},
author = {Sinha, Ayan and Lee, Justin and Li, Shuai and Barbastathis, George},
abstractNote = {Deep learning has been proven to yield reliably generalizable solutions to numerous classification and decision tasks. Here, we demonstrate for the first time to our knowledge that deep neural networks (DNNs) can be trained to solve end-to-end inverse problems in computational imaging. We experimentally built and tested a lensless imaging system where a DNN was trained to recover phase objects given their propagated intensity diffraction patterns.},
doi = {10.1364/OPTICA.4.001117},
journal = {Optica},
number = 9,
volume = 4,
place = {United States},
year = {2017},
month = {9}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1364/OPTICA.4.001117

Citation Metrics:
Cited by: 56 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Deep learning
journal, May 2015

  • LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
  • Nature, Vol. 521, Issue 7553
  • DOI: 10.1038/nature14539

Mastering the game of Go with deep neural networks and tree search
journal, January 2016

  • Silver, David; Huang, Aja; Maddison, Chris J.
  • Nature, Vol. 529, Issue 7587
  • DOI: 10.1038/nature16961

Deep Convolutional Neural Network for Inverse Problems in Imaging
journal, September 2017

  • Jin, Kyong Hwan; McCann, Michael T.; Froustey, Emmanuel
  • IEEE Transactions on Image Processing, Vol. 26, Issue 9
  • DOI: 10.1109/TIP.2017.2713099

Learning-based imaging through scattering media
journal, January 2016

  • Horisaki, Ryoichi; Takagi, Ryosuke; Tanida, Jun
  • Optics Express, Vol. 24, Issue 13
  • DOI: 10.1364/OE.24.013738

Robust Image Deblurring With an Inaccurate Blur Kernel
journal, April 2012


Neural networks and physical systems with emergent collective computational abilities.
journal, April 1982

  • Hopfield, J. J.
  • Proceedings of the National Academy of Sciences, Vol. 79, Issue 8
  • DOI: 10.1073/pnas.79.8.2554

Optical implementation of the Hopfield model for two-dimensional associative memory
journal, January 1988

  • Jang, Ju-Seog; Jung, Su-Won; Lee, Soo-Young
  • Optics Letters, Vol. 13, Issue 3
  • DOI: 10.1364/OL.13.000248

Human-level control through deep reinforcement learning
journal, February 2015

  • Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David
  • Nature, Vol. 518, Issue 7540
  • DOI: 10.1038/nature14236

Reconstruction of an object from the modulus of its Fourier transform
journal, January 1978


Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization
journal, February 2003

  • Donoho, D. L.; Elad, M.
  • Proceedings of the National Academy of Sciences, Vol. 100, Issue 5, p. 2197-2202
  • DOI: 10.1073/pnas.0437847100

Digital Image Formation from Electronically Detected Holograms
journal, August 1967

  • Goodman, J. W.; Lawrence, R. W.
  • Applied Physics Letters, Vol. 11, Issue 3
  • DOI: 10.1063/1.1755043

An Introduction To Compressive Sampling
journal, March 2008


ImageNet Large Scale Visual Recognition Challenge
journal, April 2015

  • Russakovsky, Olga; Deng, Jia; Su, Hao
  • International Journal of Computer Vision, Vol. 115, Issue 3
  • DOI: 10.1007/s11263-015-0816-y

Learning approach to optical tomography
journal, January 2015

  • Kamilov, Ulugbek S.; Papadopoulos, Ioannis N.; Shoreh, Morteza H.
  • Optica, Vol. 2, Issue 6
  • DOI: 10.1364/OPTICA.2.000517

Cognitron: A self-organizing multilayered neural network
journal, January 1975


Learning representations by back-propagating errors
journal, October 1986

  • Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J.
  • Nature, Vol. 323, Issue 6088
  • DOI: 10.1038/323533a0

Deterministic phase retrieval: a Green’s function solution
journal, January 1983

  • Teague, Michael Reed
  • Journal of the Optical Society of America, Vol. 73, Issue 11
  • DOI: 10.1364/JOSA.73.001434

Phase imaging by the transport equation of intensity
journal, February 1984


Measuring the Transmission Matrix in Optics: An Approach to the Study and Control of Light Propagation in Disordered Media
journal, March 2010


    Works referencing / citing this record:

    Deep iterative reconstruction for phase retrieval
    journal, January 2019

    • Işıl, Çağatay; Oktem, Figen S.; Koç, Aykut
    • Applied Optics, Vol. 58, Issue 20
    • DOI: 10.1364/ao.58.005422

    Phaseless coherent and incoherent microwave ghost imaging with dynamic metasurface apertures
    journal, January 2018

    • Diebold, Aaron V.; Imani, Mohammadreza F.; Sleasman, Timothy
    • Optica, Vol. 5, Issue 12
    • DOI: 10.1364/optica.5.001529

    Real-time particle pollution sensing using machine learning
    journal, January 2018

    • Grant-Jacob, James A.; Mackay, Benita S.; Baker, James A. G.
    • Optics Express, Vol. 26, Issue 21
    • DOI: 10.1364/oe.26.027237

    Deep iterative reconstruction for phase retrieval
    journal, January 2019

    • Işıl, Çağatay; Oktem, Figen S.; Koç, Aykut
    • Applied Optics, Vol. 58, Issue 20
    • DOI: 10.1364/ao.58.005422

    All-optical machine learning using diffractive deep neural networks
    journal, July 2018


    Optimal physical preprocessing for example-based super-resolution
    journal, January 2018


    Deep learning optical-sectioning method
    journal, January 2018

    • Zhang, Xiaoyu; Chen, Yifan; Ning, Kefu
    • Optics Express, Vol. 26, Issue 23
    • DOI: 10.1364/oe.26.030762

    Quantitative phase imaging in biomedicine
    journal, September 2018

    • Park, YongKeun; Depeursinge, Christian; Popescu, Gabriel
    • Nature Photonics, Vol. 12, Issue 10
    • DOI: 10.1038/s41566-018-0253-x

    Low Photon Count Phase Retrieval Using Deep Learning
    journal, December 2018


    Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging
    journal, January 2019


    Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery
    journal, January 2018


    Intelligent metasurface imager and recognizer
    journal, October 2019


    Reliable deep-learning-based phase imaging with uncertainty quantification
    journal, January 2019


    eHoloNet: a learning-based end-to-end approach for in-line digital holographic reconstruction
    journal, January 2018


    Deep hybrid scattering image learning
    journal, January 2019

    • Yang, Mu; Liu, Zheng-Hao; Cheng, Ze-Di
    • Journal of Physics D: Applied Physics, Vol. 52, Issue 11
    • DOI: 10.1088/1361-6463/aafa3c

    Direct imaging of molecular rotation with high-order-harmonic generation
    journal, May 2019


    Resolution-enhanced light field displays by recombining subpixels across elemental images
    journal, January 2019


    Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media
    journal, January 2018


    Modal classification in optical waveguides using deep learning
    journal, December 2018


    RedCap: residual encoder-decoder capsule network for holographic image reconstruction
    journal, January 2020

    • Zeng, Tianjiao; So, Hayden K. -H.; Lam, Edmund Y.
    • Optics Express, Vol. 28, Issue 4
    • DOI: 10.1364/oe.383350

    Low Photon Count Phase Retrieval Using Deep Learning
    journal, December 2018


    A path to high-quality imaging through disordered optical fibers: a review
    journal, January 2019

    • Zhao, Jian; Peysokhan, Mostafa; Antonio-Lopez, Jose Enrique
    • Applied Optics, Vol. 58, Issue 13
    • DOI: 10.1364/ao.58.000d50

    PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning
    journal, February 2019


    Light scattering control in transmission and reflection with neural networks
    journal, January 2018

    • Turpin, Alex; Vishniakou, Ivan; Seelig, Johannes d.
    • Optics Express, Vol. 26, Issue 23
    • DOI: 10.1364/oe.26.030911

    Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery
    journal, January 2018


    DiffuserCam: lensless single-exposure 3D imaging
    journal, December 2017


    Computational cytometer based on magnetically modulated coherent imaging and deep learning
    journal, October 2019

    • Zhang, Yibo; Ouyang, Mengxing; Ray, Aniruddha
    • Light: Science & Applications, Vol. 8, Issue 1
    • DOI: 10.1038/s41377-019-0203-5

    Efficient and accurate inversion of multiple scattering with deep learning
    journal, January 2018

    • Sun, Yu; Xia, Zhihao; Kamilov, Ulugbek S.
    • Optics Express, Vol. 26, Issue 11
    • DOI: 10.1364/oe.26.014678

    DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks
    journal, January 2019

    • Gedalin, Daniel; Oiknine, Yaniv; Stern, Adrian
    • Optics Express, Vol. 27, Issue 24
    • DOI: 10.1364/oe.27.035811

    Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram
    journal, March 2019


    Deep learning in holography and coherent imaging
    journal, September 2019


    Imaging through glass diffusers using densely connected convolutional networks
    journal, January 2018


    Computational complex optical field imaging using a designed metasurface diffuser
    journal, January 2018


    Real-time particle pollution sensing using machine learning
    journal, January 2018

    • Grant-Jacob, James A.; Mackay, Benita S.; Baker, James A. G.
    • Optics Express, Vol. 26, Issue 21
    • DOI: 10.1364/oe.26.027237

    High-resolution limited-angle phase tomography of dense layered objects using deep neural networks
    journal, September 2019

    • Goy, Alexandre; Rughoobur, Girish; Li, Shuai
    • Proceedings of the National Academy of Sciences, Vol. 116, Issue 40
    • DOI: 10.1073/pnas.1821378116

    Real-time coherent diffraction inversion using deep generative networks
    journal, November 2018

    • Cherukara, Mathew J.; Nashed, Youssef S. G.; Harder, Ross J.
    • Scientific Reports, Vol. 8, Issue 1
    • DOI: 10.1038/s41598-018-34525-1

    DiffuserCam: lensless single-exposure 3D imaging
    journal, December 2017


    One-step robust deep learning phase unwrapping
    journal, January 2019


    Spectral pre-modulation of training examples enhances the spatial resolution of the phase extraction neural network (PhENN)
    journal, January 2018


    Three-dimensional localization microscopy using deep learning
    journal, January 2018


    Multimode optical fiber transmission with a deep learning network
    journal, October 2018

    • Rahmani, Babak; Loterie, Damien; Konstantinou, Georgia
    • Light: Science & Applications, Vol. 7, Issue 1
    • DOI: 10.1038/s41377-018-0074-1

    Regularized reconstruction of absorbing and phase objects from a single in-line hologram, application to fluid mechanics and micro-biology
    journal, January 2018

    • Jolivet, Frédéric; Momey, Fabien; Denis, Loïc
    • Optics Express, Vol. 26, Issue 7
    • DOI: 10.1364/oe.26.008923

    Deep learning approach for Fourier ptychography microscopy
    journal, January 2018


    Deep learning the high variability and randomness inside multimode fibers
    journal, January 2019


    Imaging through glass diffusers using densely connected convolutional networks
    journal, January 2018


    Computational complex optical field imaging using a designed metasurface diffuser
    journal, January 2018


    Parallel lensless compressive imaging via deep convolutional neural networks
    journal, January 2018


    Deep-learning-based ghost imaging
    journal, December 2017


    On the use of deep learning for computational imaging
    journal, January 2019


    Optical inspection of nanoscale structures using a novel machine learning based synthetic image generation algorithm
    journal, January 2019

    • Purandare, Sanyogita; Zhu, Jinlong; Zhou, Renjie
    • Optics Express, Vol. 27, Issue 13
    • DOI: 10.1364/oe.27.017743

    Learned Integrated Sensing Pipeline: Reconfigurable Metasurface Transceivers as Trainable Physical Layer in an Artificial Neural Network
    journal, December 2019

    • Hougne, Philipp; Imani, Mohammadreza F.; Diebold, Aaron V.
    • Advanced Science, Vol. 7, Issue 3
    • DOI: 10.1002/advs.201901913

    Quantum-inspired computational imaging
    journal, August 2018

    • Altmann, Yoann; McLaughlin, Stephen; Padgett, Miles J.
    • Science, Vol. 361, Issue 6403
    • DOI: 10.1126/science.aat2298

    Deep learning wavefront sensing
    journal, January 2019

    • Nishizaki, Yohei; Valdivia, Matias; Horisaki, Ryoichi
    • Optics Express, Vol. 27, Issue 1
    • DOI: 10.1364/oe.27.000240

    Deep learning approach for Fourier ptychography microscopy
    journal, January 2018


    Neural network based design of metagratings
    journal, June 2018

    • Inampudi, Sandeep; Mosallaei, Hossein
    • Applied Physics Letters, Vol. 112, Issue 24
    • DOI: 10.1063/1.5033327

    Brief review on learning-based methods for optical tomography
    journal, November 2019

    • Zhang, Lin; Zhang, Guanglei
    • Journal of Innovative Optical Health Sciences, Vol. 12, Issue 06
    • DOI: 10.1142/s1793545819300118

    Cryptanalysis of random-phase-encoding-based optical cryptosystem via deep learning
    journal, January 2019


    High-throughput, volumetric quantitative phase imaging with multiplexed intensity diffraction tomography
    journal, January 2019


    Analysis of Phase-Extraction Neural Network (PhENN) performance for lensless quantitative phase imaging
    conference, March 2019

    • Li, Shuai; Barbastathis, George; Goy, Alexandre
    • Quantitative Phase Imaging V
    • DOI: 10.1117/12.2513310

    Artificial intelligence-assisted light control and computational imaging through scattering media
    journal, July 2019

    • Cheng, Shengfu; Li, Huanhao; Luo, Yunqi
    • Journal of Innovative Optical Health Sciences, Vol. 12, Issue 04
    • DOI: 10.1142/s1793545819300064

    Deep spectral learning for label-free optical imaging oximetry with uncertainty quantification
    journal, November 2019


    Machine Learning Based Localization and Classification with Atomic Magnetometers
    journal, January 2018


    Fringe pattern analysis using deep learning
    journal, February 2019


    Design of task-specific optical systems using broadband diffractive neural networks
    journal, December 2019


    Reliable deep-learning-based phase imaging with uncertainty quantification
    journal, January 2019


    Improving Imaging Quality of Real-time Fourier Single-pixel Imaging via Deep Learning
    journal, September 2019

    • Rizvi, Saad; Cao, Jie; Zhang, Kaiyu
    • Sensors, Vol. 19, Issue 19
    • DOI: 10.3390/s19194190

    Learning to see through multimode fibers
    journal, January 2018


    One-step robust deep learning phase unwrapping
    journal, January 2019


    Neural network identification of people hidden from view with a single-pixel, single-photon detector
    journal, August 2018

    • Caramazza, Piergiorgio; Boccolini, Alessandro; Buschek, Daniel
    • Scientific Reports, Vol. 8, Issue 1
    • DOI: 10.1038/s41598-018-30390-0

    Deep learning-based super-resolution in coherent imaging systems
    journal, March 2019


    Phase recovery and holographic image reconstruction using deep learning in neural networks
    journal, October 2017

    • Rivenson, Yair; Zhang, Yibo; Günaydın, Harun
    • Light: Science & Applications, Vol. 7, Issue 2
    • DOI: 10.1038/lsa.2017.141

    Machine learning for improved image-based wavefront sensing
    journal, January 2018


    Y-Net: a one-to-two deep learning framework for digital holographic reconstruction
    journal, January 2019

    • Wang, Kaiqiang; Dou, Jiazhen; Kemao, Qian
    • Optics Letters, Vol. 44, Issue 19
    • DOI: 10.1364/ol.44.004765

    Regularized reconstruction of absorbing and phase objects from a single in-line hologram, application to fluid mechanics and micro-biology
    journal, January 2018

    • Jolivet, Frédéric; Momey, Fabien; Denis, Loïc
    • Optics Express, Vol. 26, Issue 7
    • DOI: 10.1364/oe.26.008923

    cellSTORM—Cost-effective super-resolution on a cellphone using dSTORM
    journal, January 2019