Deep learning at the edge enables real-time streaming ptychographic imaging

Babu, Anakha V.; Zhou, Tao; Kandel, Saugat; Bicer, Tekin; Liu, Zhengchun; Judge, William; Ching, Daniel J.; Jiang, Yi; Veseli, Sinisa; Henke, Steven; Chard, Ryan; Yao, Yudong; Sirazitdinova, Ekaterina; Gupta, Geetika; Holt, Martin V.; Foster, Ian T.; Miceli, Antonino; Cherukara, Mathew J.

doi:10.1038/s41467-023-41496-z

Title: Deep learning at the edge enables real-time streaming ptychographic imaging

Journal Article · 02 November 2023 · Nature Communications

DOI: https://doi.org/10.1038/s41467-023-41496-z · OSTI ID:2204731

Babu, Anakha V.; Zhou, Tao;

;

; Liu, Zhengchun; Judge, William;

;

; Chard, Ryan; Yao, Yudong; Sirazitdinova, Ekaterina; Gupta, Geetika;

;

Abstract Coherent imaging techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent imaging methods like ptychography are poised to revolutionize nanoscale materials characterization. However, these advancements are accompanied by significant increase in data and compute needs, which precludes real-time imaging, feedback and decision-making capabilities with conventional approaches. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the oversampling constraints, allowing low-dose imaging using orders of magnitude less data than required by traditional methods.

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 2204731

Alternate ID(s):: OSTI ID: 2470142; OSTI ID: 2538607

Journal Information:: Nature Communications, Journal Name: Nature Communications Journal Issue: 1 Vol. 14; ISSN 2041-1723

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (44)

Measuring Three-Dimensional Strain and Structural Defects in a Single InGaAs Nanowire Using Coherent X-ray Multiangle Bragg Projection Ptychography Hill, Megan O.; Calvo-Almazan, Irene; Allain, Marc Nano Letters, Vol. 18, Issue 2 https://doi.org/10.1021/acs.nanolett.7b04024	journal	January 2018
Differential programming enabled functional imaging with Lorentz transmission electron microscopy Zhou, Tao; Cherukara, Mathew; Phatak, Charudatta npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00600-x	journal	September 2021
Image Quality Assessment: From Error Visibility to Structural Similarity Wang, Z.; Bovik, A. C.; Sheikh, H. R. IEEE Transactions on Image Processing, Vol. 13, Issue 4 https://doi.org/10.1109/TIP.2003.819861	journal	April 2004
Ptychographic X-ray computed tomography at the nanoscale Dierolf, Martin; Menzel, Andreas; Thibault, Pierre Nature, Vol. 467, Issue 7314 https://doi.org/10.1038/nature09419	journal	September 2010
Single-shot ptychography Sidorenko, Pavel; Cohen, Oren Optica, Vol. 3, Issue 1 https://doi.org/10.1364/OPTICA.3.000009	journal	December 2015
JUNGFRAU detector for brighter x-ray sources: Solutions for IT and data science challenges in macromolecular crystallography Leonarski, Filip; Mozzanica, Aldo; Brückner, Martin Structural Dynamics, Vol. 7, Issue 1 https://doi.org/10.1063/1.5143480	journal	January 2020
Deep neural networks in single-shot ptychography Wengrowicz, Omri; Peleg, Or; Zahavy, Tom Optics Express, Vol. 28, Issue 12 https://doi.org/10.1364/OE.393961	journal	January 2020
Scalable and accurate multi-GPU-based image reconstruction of large-scale ptychography data Yu, Xiaodong; Nikitin, Viktor; Ching, Daniel J. Scientific Reports, Vol. 12, Issue 1 https://doi.org/10.1038/s41598-022-09430-3	journal	March 2022
Achieving high spatial resolution in a large field-of-view using lensless x-ray imaging Jiang, Yi; Deng, Junjing; Yao, Yudong Applied Physics Letters, Vol. 119, Issue 12 https://doi.org/10.1063/5.0067197	journal	September 2021
Very-High Dynamic Range, 10,000 Frames/Second Pixel Array Detector for Electron Microscopy Philipp, Hugh T.; Tate, Mark W.; Shanks, Katherine S. Microscopy and Microanalysis, Vol. 28, Issue 2 https://doi.org/10.1017/S1431927622000174	journal	March 2022
High-Performance Ptychographic Reconstruction with Federated Facilities Bicer, Tekin; Yu, Xiaodong; Ching, Daniel J. Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation https://doi.org/10.1007/978-3-030-96498-6_10	book	January 2022
Further improvements to the ptychographical iterative engine Maiden, Andrew; Johnson, Daniel; Li, Peng Optica, Vol. 4, Issue 7 https://doi.org/10.1364/OPTICA.4.000736	journal	January 2017
An improved ptychographical phase retrieval algorithm for diffractive imaging Maiden, Andrew M.; Rodenburg, John M. Ultramicroscopy, Vol. 109, Issue 10 https://doi.org/10.1016/j.ultramic.2009.05.012	journal	September 2009
Upscaling X-ray nanoimaging to macroscopic specimens Du, Ming; Di, Zichao; Gürsoy, Dogˇa Journal of Applied Crystallography, Vol. 54, Issue 2 https://doi.org/10.1107/S1600576721000194	journal	February 2021
PyNX : high-performance computing toolkit for coherent X-ray imaging based on operators Favre-Nicolin, Vincent; Girard, Gaétan; Leake, Steven Journal of Applied Crystallography, Vol. 53, Issue 5 https://doi.org/10.1107/S1600576720010985	journal	September 2020
TomoGAN: low-dose synchrotron x-ray tomography with generative adversarial networks: discussion Liu, Zhengchun; Bicer, Tekin; Kettimuthu, Rajkumar Journal of the Optical Society of America A, Vol. 37, Issue 3 https://doi.org/10.1364/JOSAA.375595	journal	January 2020
funcX: A Federated Function Serving Fabric for Science Chard, Ryan; Babuji, Yadu; Li, Zhuozhao HPDC '20: The 29th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3369583.3392683	conference	June 2020
High Dynamic Range Pixel Array Detector for Scanning Transmission Electron Microscopy Tate, Mark W.; Purohit, Prafull; Chamberlain, Darol Microscopy and Microanalysis, Vol. 22, Issue 1 https://doi.org/10.1017/S1431927615015664	journal	January 2016
Multi-slice ptychographic tomography Li, Peng; Maiden, Andrew Scientific Reports, Vol. 8, Issue 1 https://doi.org/10.1038/s41598-018-20530-x	journal	February 2018
Cyclical Learning Rates for Training Neural Networks Smith, Leslie N. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) https://doi.org/10.1109/WACV.2017.58	conference	March 2017
Low Photon Count Phase Retrieval Using Deep Learning Goy, Alexandre; Arthur, Kwabena; Li, Shuai Physical Review Letters, Vol. 121, Issue 24 https://doi.org/10.1103/PhysRevLett.121.243902	journal	December 2018
AutoPhaseNN: unsupervised physics-aware deep learning of 3D nanoscale Bragg coherent diffraction imaging Yao, Yudong; Chan, Henry; Sankaranarayanan, Subramanian npj Computational Materials, Vol. 8, Issue 1 https://doi.org/10.1038/s41524-022-00803-w	journal	June 2022
Eiger: a single-photon counting x-ray detector Johnson, I.; Bergamaschi, A.; Billich, H. Journal of Instrumentation, Vol. 9, Issue 05 https://doi.org/10.1088/1748-0221/9/05/C05032	journal	May 2014
Three-dimensional coherent X-ray diffraction imaging via deep convolutional neural networks Wu, Longlong; Yoo, Shinjae; Suzana, Ana F. npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00644-z	journal	October 2021
PtychoNet: Fast and High Quality Phase Retrieval for Ptychography Guan, Ziqiao; Tsai, Esther; Huang, Xiaojing https://doi.org/10.2172/1599580	report	September 2019
Deep learning approach for Fourier ptychography microscopy Nguyen, Thanh; Xue, Yujia; Li, Yunzhe Optics Express, Vol. 26, Issue 20 https://doi.org/10.1364/OE.26.026470	journal	January 2018
BraggNN: fast X-ray Bragg peak analysis using deep learning Liu, Zhengchun; Sharma, Hemant; Park, Jun-Sang IUCrJ, Vol. 9, Issue 1 https://doi.org/10.1107/S2052252521011258	journal	December 2021
SHARP : a distributed GPU-based ptychographic solver Marchesini, Stefano; Krishnan, Hari; Daurer, Benedikt J. Journal of Applied Crystallography, Vol. 49, Issue 4 https://doi.org/10.1107/S1600576716008074	journal	July 2016
fairDMS: Rapid Model Training by Data and Model Reuse Ali, Ahsan; Sharma, Hemant; Kettimuthu, Rajkumar 2022 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER51413.2022.00050	conference	September 2022
On-the-fly scans for X-ray ptychography Pelz, Philipp M.; Guizar-Sicairos, Manuel; Thibault, Pierre Applied Physics Letters, Vol. 105, Issue 25 https://doi.org/10.1063/1.4904943	journal	December 2014
Electron ptychography achieves atomic-resolution limits set by lattice vibrations Chen, Zhen; Jiang, Yi; Shao, Yu-Tsun Science, Vol. 372, Issue 6544 https://doi.org/10.1126/science.abg2533	journal	May 2021
AI-enabled high-resolution scanning coherent diffraction imaging Cherukara, Mathew J.; Zhou, Tao; Nashed, Youssef Applied Physics Letters, Vol. 117, Issue 4 https://doi.org/10.1063/5.0013065	journal	July 2020
Electron ptychography of 2D materials to deep sub-ångström resolution Jiang, Yi; Chen, Zhen; Han, Yimo Nature, Vol. 559, Issue 7714 https://doi.org/10.1038/s41586-018-0298-5	journal	July 2018
The PILATUS 1M detector Broennimann, Ch.; Eikenberry, E. F.; Henrich, B. Journal of Synchrotron Radiation, Vol. 13, Issue 2 https://doi.org/10.1107/S0909049505038665	journal	February 2006
Topology-aware optimizations for multi-GPU ptychographic image reconstruction Yu, Xiaodong; Bicer, Tekin; Kettimuthu, Rajkumar ICS '21: 2021 International Conference on Supercomputing, Proceedings of the ACM International Conference on Supercomputing https://doi.org/10.1145/3447818.3460380	conference	June 2021
High-resolution non-destructive three-dimensional imaging of integrated circuits Holler, Mirko; Guizar-Sicairos, Manuel; Tsai, Esther H. R. Nature, Vol. 543, Issue 7645 https://doi.org/10.1038/nature21698	journal	March 2017
High-resolution three-dimensional structural microscopy by single-angle Bragg ptychography Hruszkewycz, S. O.; Allain, M.; Holt, M. V. Nature Materials, Vol. 16, Issue 2 https://doi.org/10.1038/nmat4798	journal	November 2016
Deep neural networks in real-time coherent diffraction imaging Harder, Ross IUCrJ, Vol. 8, Issue 1 https://doi.org/10.1107/S2052252520016590	journal	January 2021
Fourier ptychography: current applications and future promises Konda, Pavan Chandra; Loetgering, Lars; Zhou, Kevin C. Optics Express, Vol. 28, Issue 7 https://doi.org/10.1364/OE.386168	journal	March 2020
Deep-Learning Electron Diffractive Imaging Chang, Dillan J.; O’Leary, Colum M.; Su, Cong Physical Review Letters, Vol. 130, Issue 1 https://doi.org/10.1103/PhysRevLett.130.016101	journal	January 2023
Simultaneous cryo X-ray ptychographic and fluorescence microscopy of green algae Deng, Junjing; Vine, David J.; Chen, Si Proceedings of the National Academy of Sciences, Vol. 112, Issue 8 https://doi.org/10.1073/pnas.1413003112	journal	February 2015
Rapid 3D nanoscale coherent imaging via physics-aware deep learning Chan, Henry; Nashed, Youssef S. G.; Kandel, Saugat Applied Physics Reviews, Vol. 8, Issue 2 https://doi.org/10.1063/5.0031486	journal	June 2021
Linking scientific instruments and computation: Patterns, technologies, and experiences Vescovi, Rafael; Chard, Ryan; Saint, Nickolaus D. Patterns, Vol. 3, Issue 10 https://doi.org/10.1016/j.patter.2022.100606	journal	October 2022
The Velociprobe: An ultrafast hard X-ray nanoprobe for high-resolution ptychographic imaging Deng, Junjing; Preissner, Curt; Klug, Jeffrey A. Review of Scientific Instruments, Vol. 90, Issue 8 https://doi.org/10.1063/1.5103173	journal	August 2019

Similar Records

Real-time sparse-sampled Ptychographic imaging through deep neural networks

Journal Article · 2020 · OSTI ID:1713219

Cherukara, Mathew J; Zhou, Tao; Nashed, Youssef; +4 more

Real-Time Interactive 4D-STEM Phase-Contrast Imaging From Electron Event Representation Data: Less computation with the right representation

Journal Article · 2021 · IEEE Signal Processing Magazine · OSTI ID:1827921

Pelz, Philipp M.; Johnson, Ian; Ophus, Colin; +2 more

SHARP: a distributed GPU-based ptychographic solver

Journal Article · 2016 · Journal of Applied Crystallography (Online) · OSTI ID:1764514

Marchesini, Stefano; Krishnan, Hari; Daurer, Benedikt J.; +4 more

Title: Deep learning at the edge enables real-time streaming ptychographic imaging

Citation Formats

References (44)

Similar Records

Related Subjects