Rapid 3D nanoscale coherent imaging via physics-aware deep learning

Chan, Henry; Nashed, Youssef G.; Kandel, Saugat; Hruszkewycz, Stephan O.; Sankaranarayanan, Subramanian S.; Harder, Ross J.; Cherukara, Mathew J.

doi:10.1063/5.0031486

Title: Rapid 3D nanoscale coherent imaging via physics-aware deep learning

Abstract

Phase retrieval, the problem of recovering lost phase information from measured intensity alone, is an inverse problem that is widely faced in various imaging modalities ranging from astronomy to nanoscale imaging. The current process of phase recovery is iterative in nature. As a result, the image formation is time consuming and computationally expensive, precluding real-time imaging. Here, we use 3D nanoscale X-ray imaging as a representative example to develop a deep learning model to address this phase retrieval problem. We introduce 3D-CDI-NN, a deep convolutional neural network and differential programing framework trained to predict 3D structure and strain, solely from input 3D X-ray coherent scattering data. Our networks are designed to be "physics-aware" in multiple aspects; in that the physics of the X-ray scattering process is explicitly enforced in the training of the network, and the training data are drawn from atomistic simulations that are representative of the physics of the material. We further refine the neural network prediction through a physics-based optimization procedure to enable maximum accuracy at lowest computational cost. 3D-CDI-NN can invert a 3D coherent diffraction pattern to real-space structure and strain hundreds of times faster than traditional iterative phase retrieval methods. Our integrated machine learning andmore »« less

Authors:

^[1];

^[2];

^[3];

^[4];

^[1];

^[5];

^[6]

Argonne National Lab. (ANL), Argonne, IL (United States). Center for Nanoscale Materials; Univ. of Illinois, Chicago, IL (United States). Dept. of Mechanical and Industrial Engineering
SLAC National Accelerator Lab., Menlo Park, CA (United States)
Northwestern Univ., Evanston, IL (United States). Applied Physics
Argonne National Lab. (ANL), Argonne, IL (United States). Materials Science Division
Argonne National Lab. (ANL), Argonne, IL (United States). Advanced Photon Source (APS)
Argonne National Lab. (ANL), Argonne, IL (United States). Center for Nanoscale Materials; Argonne National Lab. (ANL), Argonne, IL (United States). Advanced Photon Source (APS)

Publication Date:: Mon May 17 00:00:00 EDT 2021

Research Org.:: SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States); Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Basic Energy Sciences (BES)

OSTI Identifier:: 1785971

Alternate Identifier(s):: OSTI ID: 1783342; OSTI ID: 1788382

Grant/Contract Number:: AC02-76SF00515; 34532; AC02- 06CH11357; 2018-019-N0; Award Number 34532; AC02-06CH11357

Resource Type:: Accepted Manuscript

Journal Name:: Applied Physics Reviews

Additional Journal Information:: Journal Volume: 8; Journal Issue: 2; Journal ID: ISSN 1931-9401

Publisher:: American Institute of Physics (AIP)

Country of Publication:: United States

Language:: English

Subject:: 77 NANOSCIENCE AND NANOTECHNOLOGY

Citation Formats


                    Chan, Henry, Nashed, Youssef G., Kandel, Saugat, Hruszkewycz, Stephan O., Sankaranarayanan, Subramanian S., Harder, Ross J., and Cherukara, Mathew J. Rapid 3D nanoscale coherent imaging via physics-aware deep learning.  United States: N. p., 2021. 
Web.  doi:10.1063/5.0031486.

Copy to clipboard


                    Chan, Henry, Nashed, Youssef G., Kandel, Saugat, Hruszkewycz, Stephan O., Sankaranarayanan, Subramanian S., Harder, Ross J., & Cherukara, Mathew J. Rapid 3D nanoscale coherent imaging via physics-aware deep learning.  United States.  https://doi.org/10.1063/5.0031486

Copy to clipboard


                    Chan, Henry, Nashed, Youssef G., Kandel, Saugat, Hruszkewycz, Stephan O., Sankaranarayanan, Subramanian S., Harder, Ross J., and Cherukara, Mathew J. Mon .  
"Rapid 3D nanoscale coherent imaging via physics-aware deep learning".  United States.  https://doi.org/10.1063/5.0031486.  https://www.osti.gov/servlets/purl/1785971.

Copy to clipboard


                    
@article{osti_1785971,

  title        = {Rapid 3D nanoscale coherent imaging via physics-aware deep learning},

  author       = {Chan, Henry and Nashed, Youssef G. and Kandel, Saugat and Hruszkewycz, Stephan O. and Sankaranarayanan, Subramanian S. and Harder, Ross J. and Cherukara, Mathew J.},

  abstractNote = {Phase retrieval, the problem of recovering lost phase information from measured intensity alone, is an inverse problem that is widely faced in various imaging modalities ranging from astronomy to nanoscale imaging. The current process of phase recovery is iterative in nature. As a result, the image formation is time consuming and computationally expensive, precluding real-time imaging. Here, we use 3D nanoscale X-ray imaging as a representative example to develop a deep learning model to address this phase retrieval problem. We introduce 3D-CDI-NN, a deep convolutional neural network and differential programing framework trained to predict 3D structure and strain, solely from input 3D X-ray coherent scattering data. Our networks are designed to be "physics-aware" in multiple aspects; in that the physics of the X-ray scattering process is explicitly enforced in the training of the network, and the training data are drawn from atomistic simulations that are representative of the physics of the material. We further refine the neural network prediction through a physics-based optimization procedure to enable maximum accuracy at lowest computational cost. 3D-CDI-NN can invert a 3D coherent diffraction pattern to real-space structure and strain hundreds of times faster than traditional iterative phase retrieval methods. Our integrated machine learning and differential programing solution to the phase retrieval problem is broadly applicable across inverse problems in other application areas.},

  doi          = {10.1063/5.0031486},

  journal      = {Applied Physics Reviews},

  number       = 2,

  volume       = 8,

  place        = {United States},

  year         = {Mon May 17 00:00:00 EDT 2021},

  month        = {Mon May 17 00:00:00 EDT 2021}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1063/5.0031486

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Three-dimensional imaging of strain in a single ZnO nanorod
journal, December 2009

Newton, Marcus C.; Leake, Steven J.; Harder, Ross
Nature Materials, Vol. 9, Issue 2
DOI: 10.1038/nmat2607

Phase retrieval algorithm for JWST Flight and Testbed Telescope
conference, July 2006

Dean, Bruce H.; Aronstein, David L.; Smith, J. Scott
SPIE Astronomical Telescopes + Instrumentation, SPIE Proceedings
DOI: 10.1117/12.673569

Topological defect dynamics in operando battery nanoparticles
journal, June 2015

Ulvestad, A.; Singer, A.; Clark, J. N.
Science, Vol. 348, Issue 6241
DOI: 10.1126/science.aaa1313

Three-dimensional X-ray diffraction imaging of dislocations in polycrystalline metals under tensile loading
journal, September 2018

Cherukara, Mathew J.; Pokharel, Reeju; O’Leary, Timothy S.
Nature Communications, Vol. 9, Issue 1
DOI: 10.1038/s41467-018-06166-5

Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition
journal, October 2019

Winkler, Julia K.; Fink, Christine; Toberer, Ferdinand
JAMA Dermatology, Vol. 155, Issue 10
DOI: 10.1001/jamadermatol.2019.1735

Using automatic differentiation as a general framework for ptychographic reconstruction
journal, January 2019

Kandel, Saugat; Maddali, S.; Allain, Marc
Optics Express, Vol. 27, Issue 13
DOI: 10.1364/OE.27.018653

Beyond crystallography: Diffractive imaging using coherent x-ray light sources
journal, April 2015

Miao, J.; Ishikawa, T.; Robinson, I. K.
Science, Vol. 348, Issue 6234
DOI: 10.1126/science.aaa1394

AI-enabled high-resolution scanning coherent diffraction imaging
journal, July 2020

Cherukara, Mathew J.; Zhou, Tao; Nashed, Youssef
Applied Physics Letters, Vol. 117, Issue 4
DOI: 10.1063/5.0013065

Phase recovery and holographic image reconstruction using deep learning in neural networks
journal, October 2017

Rivenson, Yair; Zhang, Yibo; Günaydın, Harun
Light: Science & Applications, Vol. 7, Issue 2
DOI: 10.1038/lsa.2017.141

Bragg coherent diffractive imaging of single-grain defect dynamics in polycrystalline films
journal, May 2017

Yau, Allison; Cha, Wonsuk; Kanan, Matthew W.
Science, Vol. 356, Issue 6339
DOI: 10.1126/science.aam6168

Ultrafast Three-Dimensional X-ray Imaging of Deformation Modes in ZnO Nanocrystals
journal, January 2017

Cherukara, Mathew J.; Sasikumar, Kiran; Cha, Wonsuk
Nano Letters, Vol. 17, Issue 2
DOI: 10.1021/acs.nanolett.6b04652

Fast Parallel Algorithms for Short-Range Molecular Dynamics
journal, March 1995

Plimpton, Steve
Journal of Computational Physics, Vol. 117, Issue 1
DOI: 10.1006/jcph.1995.1039

Image reconstruction by domain-transform manifold learning
journal, March 2018

Zhu, Bo; Liu, Jeremiah Z.; Cauley, Stephen F.
Nature, Vol. 555, Issue 7697
DOI: 10.1038/nature25988

Phase retrieval algorithms: a comparison
journal, January 1982

Fienup, J. R.
Applied Optics, Vol. 21, Issue 15
DOI: 10.1364/AO.21.002758

Three-Dimensional Study of the Vector Potential of Magnetic Structures
journal, June 2010

Phatak, Charudatta; Petford-Long, Amanda K.; De Graef, Marc
Physical Review Letters, Vol. 104, Issue 25
DOI: 10.1103/PhysRevLett.104.253901

3D lattice distortions and defect structures in ion-implanted nano-crystals
journal, April 2017

Hofmann, Felix; Tarleton, Edmund; Harder, Ross J.
Scientific Reports, Vol. 7, Issue 1
DOI: 10.1038/srep45993

Complex imaging of phase domains by deep neural networks
journal, January 2021

Wu, Longlong; Juhas, Pavol; Yoo, Shinjae
IUCrJ, Vol. 8, Issue 1
DOI: 10.1107/S2052252520013780

Sparsity-based single-shot subwavelength coherent diffractive imaging
journal, April 2012

Szameit, A.; Shechtman, Y.; Osherovich, E.
Nature Materials, Vol. 11, Issue 5
DOI: 10.1038/nmat3289

Identifying Defects with Guided Algorithms in Bragg Coherent Diffractive Imaging
journal, August 2017

Ulvestad, A.; Nashed, Y.; Beutier, G.
Scientific Reports, Vol. 7, Issue 1
DOI: 10.1038/s41598-017-09582-7

High-resolution three-dimensional structural microscopy by single-angle Bragg ptychography
journal, November 2016

Hruszkewycz, S. O.; Allain, M.; Holt, M. V.
Nature Materials, Vol. 16, Issue 2
DOI: 10.1038/nmat4798

In Situ 3D Imaging of Catalysis Induced Strain in Gold Nanoparticles
journal, July 2016

Ulvestad, Andrew; Sasikumar, Kiran; Kim, Jong Woo
The Journal of Physical Chemistry Letters, Vol. 7, Issue 15
DOI: 10.1021/acs.jpclett.6b01038

Active site localization of methane oxidation on Pt nanocrystals
journal, August 2018

Kim, Dongjin; Chung, Myungwoo; Carnis, Jerome
Nature Communications, Vol. 9, Issue 1
DOI: 10.1038/s41467-018-05464-2

Ultrafast Three-Dimensional Imaging of Lattice Dynamics in Individual Gold Nanocrystals
journal, May 2013

Clark, J. N.; Beitra, L.; Xiong, G.
Science, Vol. 341, Issue 6141
DOI: 10.1126/science.1236034

On resizing images in the dct domain
conference, January 2004

Salazar, C. L.; Tran, T. D.
2004 International Conference on Image Processing, 2004. ICIP '04.
DOI: 10.1109/ICIP.2004.1421685

Real-time coherent diffraction inversion using deep generative networks
journal, November 2018

Cherukara, Mathew J.; Nashed, Youssef S. G.; Harder, Ross J.
Scientific Reports, Vol. 8, Issue 1
DOI: 10.1038/s41598-018-34525-1

Coherent lensless X-ray imaging
journal, November 2010

Chapman, Henry N.; Nugent, Keith A.
Nature Photonics, Vol. 4, Issue 12
DOI: 10.1038/nphoton.2010.240

Ultrafast Three-Dimensional Integrated Imaging of Strain in Core/Shell Semiconductor/Metal Nanostructures
journal, November 2017

Cherukara, Mathew J.; Sasikumar, Kiran; DiChiara, Anthony
Nano Letters, Vol. 17, Issue 12
DOI: 10.1021/acs.nanolett.7b03823

Training Deep Neural Networks for the Inverse Design of Nanophotonic Structures
journal, January 2018

Liu, Dianjing; Tan, Yixuan; Khoram, Erfan
ACS Photonics, Vol. 5, Issue 4
DOI: 10.1021/acsphotonics.7b01377

X-ray image reconstruction from a diffraction pattern alone
journal, October 2003

Marchesini, S.; He, H.; Chapman, H. N.
Physical Review B, Vol. 68, Issue 14
DOI: 10.1103/PhysRevB.68.140101

Similar Records in DOE PAGES and OSTI.GOV collections:

2022 AI Testbed Expeditions Report

Technical Report Vishwanath, Venkat ; Emani, Murali ; Taylor, Valerie ; ...

By exploiting the coherent properties of a light source, coherent diffraction imaging (CDI) is able to obtain the sample image at a nanoscale resolution using the measured diffraction pattern. Bragg Coherent Diffraction Imaging (BCDI) has become valuable for recovering the displacement and strain field of crystals, providing a valuable tool in material science and solid-state physics. X-ray ptychography is another emerging CDI technique that can produce a high-resolution image of the extended sample and has become popular in many research areas (e.g., materials science, biology, electronics, and optics characterization). CDI including BCDI and ptychography has become an established technique inmore »« less
https://doi.org/10.2172/2217141

Full Text Available
AutoPhaseNN: unsupervised physics-aware deep learning of 3D nanoscale Bragg coherent diffraction imaging

Journal Article Yao, Yudong ; Chan, Henry ; Sankaranarayanan, Subramanian ; ... - npj Computational Materials

Abstract The problem of phase retrieval underlies various imaging methods from astronomy to nanoscale imaging. Traditional phase retrieval methods are iterative and are therefore computationally expensive. Deep learning (DL) models have been developed to either provide learned priors or completely replace phase retrieval. However, such models require vast amounts of labeled data, which can only be obtained through simulation or performing computationally prohibitive phase retrieval on experimental datasets. Using 3D X-ray Bragg coherent diffraction imaging (BCDI) as a representative technique, we demonstrate AutoPhaseNN, a DL-based approach which learns to solve the phase problem without labeled data. By incorporating the imagingmore »« less
https://doi.org/10.1038/s41524-022-00803-w
Three-dimensional coherent X-ray diffraction imaging via deep convolutional neural networks

Journal Article Wu, Longlong ; Yoo, Shinjae ; Suzana, Ana F. ; ... - npj Computational Materials

Abstract As a critical component of coherent X-ray diffraction imaging (CDI), phase retrieval has been extensively applied in X-ray structural science to recover the 3D morphological information inside measured particles. Despite meeting all the oversampling requirements of Sayre and Shannon, current phase retrieval approaches still have trouble achieving a unique inversion of experimental data in the presence of noise. Here, we propose to overcome this limitation by incorporating a 3D Machine Learning (ML) model combining (optional) supervised learning with transfer learning. The trained ML model can rapidly provide an immediate result with high accuracy which could benefit real-time experiments, andmore »« less
https://doi.org/10.1038/s41524-021-00644-z
Physics constrained unsupervised deep learning for rapid, high resolution scanning coherent diffraction reconstruction

Journal Article Hoidn, Oliver ; Mishra, Aashwin Ananda ; Mehta, Apurva - Scientific Reports

By circumventing the resolution limitations of optics, coherent diffractive imaging (CDI) and ptychography are making their way into scientific fields ranging from X-ray imaging to astronomy. Yet, the need for time consuming iterative phase recovery hampers real-time imaging. While supervised deep learning strategies have increased reconstruction speed, they sacrifice image quality. Furthermore, these methods’ demand for extensive labeled training data is experimentally burdensome. Here, we propose an unsupervised physics-informed neural network reconstruction method, PtychoPINN, that retains the factor of 100-to-1000 speedup of deep learning-based reconstruction while improving reconstruction quality by combining the diffraction forward map with real-space constraints from overlappingmore »« less
https://doi.org/10.1038/s41598-023-48351-7

Full Text Available
Real-time coherent diffraction inversion using deep generative networks

Journal Article Cherukara, Mathew J. ; Nashed, Youssef S. G. ; Harder, Ross J. - Scientific Reports

Abstract Phase retrieval, or the process of recovering phase information in reciprocal space to reconstruct images from measured intensity alone, is the underlying basis to a variety of imaging applications including coherent diffraction imaging (CDI). Typical phase retrieval algorithms are iterative in nature, and hence, are time-consuming and computationally expensive, making real-time imaging a challenge. Furthermore, iterative phase retrieval algorithms struggle to converge to the correct solution especially in the presence of strong phase structures. In this work, we demonstrate the training and testing of CDI NN, a pair of deep deconvolutional networks trained to predict structure and phase inmore »« less
Cited by 36
https://doi.org/10.1038/s41598-018-34525-1

Similar Records

Title: Rapid 3D nanoscale coherent imaging via physics-aware deep learning

Abstract

Citation Formats

Three-dimensional imaging of strain in a single ZnO nanorod journal, December 2009

Phase retrieval algorithm for JWST Flight and Testbed Telescope conference, July 2006

Topological defect dynamics in operando battery nanoparticles journal, June 2015

Three-dimensional X-ray diffraction imaging of dislocations in polycrystalline metals under tensile loading journal, September 2018

Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition journal, October 2019

Using automatic differentiation as a general framework for ptychographic reconstruction journal, January 2019

Beyond crystallography: Diffractive imaging using coherent x-ray light sources journal, April 2015

AI-enabled high-resolution scanning coherent diffraction imaging journal, July 2020

Phase recovery and holographic image reconstruction using deep learning in neural networks journal, October 2017

Bragg coherent diffractive imaging of single-grain defect dynamics in polycrystalline films journal, May 2017

Ultrafast Three-Dimensional X-ray Imaging of Deformation Modes in ZnO Nanocrystals journal, January 2017

Fast Parallel Algorithms for Short-Range Molecular Dynamics journal, March 1995

Image reconstruction by domain-transform manifold learning journal, March 2018

Phase retrieval algorithms: a comparison journal, January 1982

Three-Dimensional Study of the Vector Potential of Magnetic Structures journal, June 2010

3D lattice distortions and defect structures in ion-implanted nano-crystals journal, April 2017

Complex imaging of phase domains by deep neural networks journal, January 2021

Sparsity-based single-shot subwavelength coherent diffractive imaging journal, April 2012

Identifying Defects with Guided Algorithms in Bragg Coherent Diffractive Imaging journal, August 2017

High-resolution three-dimensional structural microscopy by single-angle Bragg ptychography journal, November 2016

In Situ 3D Imaging of Catalysis Induced Strain in Gold Nanoparticles journal, July 2016

Active site localization of methane oxidation on Pt nanocrystals journal, August 2018

Ultrafast Three-Dimensional Imaging of Lattice Dynamics in Individual Gold Nanocrystals journal, May 2013

On resizing images in the dct domain conference, January 2004

Real-time coherent diffraction inversion using deep generative networks journal, November 2018

Coherent lensless X-ray imaging journal, November 2010

Ultrafast Three-Dimensional Integrated Imaging of Strain in Core/Shell Semiconductor/Metal Nanostructures journal, November 2017

Training Deep Neural Networks for the Inverse Design of Nanophotonic Structures journal, January 2018

X-ray image reconstruction from a diffraction pattern alone journal, October 2003

Three-dimensional imaging of strain in a single ZnO nanorod
journal, December 2009

Phase retrieval algorithm for JWST Flight and Testbed Telescope
conference, July 2006

Topological defect dynamics in operando battery nanoparticles
journal, June 2015

Three-dimensional X-ray diffraction imaging of dislocations in polycrystalline metals under tensile loading
journal, September 2018

Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition
journal, October 2019

Using automatic differentiation as a general framework for ptychographic reconstruction
journal, January 2019

Beyond crystallography: Diffractive imaging using coherent x-ray light sources
journal, April 2015

AI-enabled high-resolution scanning coherent diffraction imaging
journal, July 2020

Phase recovery and holographic image reconstruction using deep learning in neural networks
journal, October 2017

Bragg coherent diffractive imaging of single-grain defect dynamics in polycrystalline films
journal, May 2017

Ultrafast Three-Dimensional X-ray Imaging of Deformation Modes in ZnO Nanocrystals
journal, January 2017

Fast Parallel Algorithms for Short-Range Molecular Dynamics
journal, March 1995

Image reconstruction by domain-transform manifold learning
journal, March 2018

Phase retrieval algorithms: a comparison
journal, January 1982

Three-Dimensional Study of the Vector Potential of Magnetic Structures
journal, June 2010

3D lattice distortions and defect structures in ion-implanted nano-crystals
journal, April 2017

Complex imaging of phase domains by deep neural networks
journal, January 2021

Sparsity-based single-shot subwavelength coherent diffractive imaging
journal, April 2012

Identifying Defects with Guided Algorithms in Bragg Coherent Diffractive Imaging
journal, August 2017

High-resolution three-dimensional structural microscopy by single-angle Bragg ptychography
journal, November 2016

In Situ 3D Imaging of Catalysis Induced Strain in Gold Nanoparticles
journal, July 2016

Active site localization of methane oxidation on Pt nanocrystals
journal, August 2018

Ultrafast Three-Dimensional Imaging of Lattice Dynamics in Individual Gold Nanocrystals
journal, May 2013

On resizing images in the dct domain
conference, January 2004

Real-time coherent diffraction inversion using deep generative networks
journal, November 2018

Coherent lensless X-ray imaging
journal, November 2010

Ultrafast Three-Dimensional Integrated Imaging of Strain in Core/Shell Semiconductor/Metal Nanostructures
journal, November 2017

Training Deep Neural Networks for the Inverse Design of Nanophotonic Structures
journal, January 2018

X-ray image reconstruction from a diffraction pattern alone
journal, October 2003