Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

Khan, Abid; Lee, Chia-Hao; Huang, Pinshane Y.; Clark, Bryan K.

doi:10.1038/s41524-023-01042-3

Title: Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

Journal Article · 28 May 2023 · npj Computational Materials

DOI: https://doi.org/10.1038/s41524-023-01042-3 · OSTI ID:1975560

Khan, Abid;

;

; Clark, Bryan K.

Abstract The rise of automation and machine learning (ML) in electron microscopy has the potential to revolutionize materials research through autonomous data collection and processing. A significant challenge lies in developing ML models that rapidly generalize to large data sets under varying experimental conditions. We address this by employing a cycle generative adversarial network (CycleGAN) with a reciprocal space discriminator, which augments simulated data with realistic spatial frequency information. This allows the CycleGAN to generate images nearly indistinguishable from real data and provide labels for ML applications. We showcase our approach by training a fully convolutional network (FCN) to identify single atom defects in a 4.5 million atom data set, collected using automated acquisition in an aberration-corrected scanning transmission electron microscope (STEM). Our method produces adaptable FCNs that can adjust to dynamically changing experimental variables with minimal intervention, marking a crucial step towards fully autonomous harnessing of microscopy big data.

View Journal Article

Cite

Export

Save

Research Organization:: University of Illinois Urbana-Champaign, IL (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Basic Energy Sciences (BES). Materials Sciences & Engineering Division (MSE)

Grant/Contract Number:: SC0020190

OSTI ID:: 1975560

Journal Information:: npj Computational Materials, Journal Name: npj Computational Materials Journal Issue: 1 Vol. 9; ISSN 2057-3960

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (49)

A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images Madsen, Jacob; Liu, Pei; Kling, Jens Advanced Theory and Simulations, Vol. 1, Issue 8 https://doi.org/10.1002/adts.201800037	journal	July 2018
Improving Shape Deformation in Unsupervised Image-to-Image Translation Gokaslan, Aaron; Ramanujan, Vivek; Ritchie, Daniel Computer Vision – ECCV 2018 https://doi.org/10.1007/978-3-030-01258-8_40	book	January 2018
Advanced Computing in Electron Microscopy Kirkland, Earl J. Springer Cham https://doi.org/10.1007/978-3-030-33260-0	book	January 2020
The Fréchet distance between multivariate normal distributions Dowson, D. C.; Landau, B. V. Journal of Multivariate Analysis, Vol. 12, Issue 3 https://doi.org/10.1016/0047-259X(82)90077-X	journal	September 1982
Pros and cons of GAN evaluation measures Borji, Ali Computer Vision and Image Understanding, Vol. 179 https://doi.org/10.1016/j.cviu.2018.10.009	journal	February 2019
Carbon contamination in scanning transmission electron microscopy and its impact on phase-plate applications Hettler, Simon; Dries, Manuel; Hermann, Peter Micron, Vol. 96 https://doi.org/10.1016/j.micron.2017.02.002	journal	May 2017
Correcting scanning instabilities from images of periodic structures Braidy, Nadi; Le Bouar, Yann; Lazar, Sorin Ultramicroscopy, Vol. 118 https://doi.org/10.1016/j.ultramic.2012.04.001	journal	July 2012
Control of radiation damage in the TEM Egerton, R. F. Ultramicroscopy, Vol. 127 https://doi.org/10.1016/j.ultramic.2012.07.006	journal	April 2013
Correcting nonlinear drift distortion of scanning probe and scanning transmission electron microscopies from image pairs with orthogonal scan directions Ophus, Colin; Ciston, Jim; Nelson, Chris T. Ultramicroscopy, Vol. 162 https://doi.org/10.1016/j.ultramic.2015.12.002	journal	March 2016
A deep convolutional neural network to analyze position averaged convergent beam electron diffraction patterns Xu, W.; LeBeau, J. M. Ultramicroscopy, Vol. 188 https://doi.org/10.1016/j.ultramic.2018.03.004	journal	May 2018
Image registration of low signal-to-noise cryo-STEM data Savitzky, Benjamin H.; El Baggari, Ismail; Clement, Colin B. Ultramicroscopy, Vol. 191 https://doi.org/10.1016/j.ultramic.2018.04.008	journal	August 2018
Theoretical framework of statistical noise in scanning transmission electron microscopy Seki, Takehito; Ikuhara, Yuichi; Shibata, Naoya Ultramicroscopy, Vol. 193 https://doi.org/10.1016/j.ultramic.2018.06.014	journal	October 2018
Improving electron micrograph signal-to-noise with an atrous convolutional encoder-decoder Ede, Jeffrey M.; Beanland, Richard Ultramicroscopy, Vol. 202 https://doi.org/10.1016/j.ultramic.2019.03.017	journal	July 2019
Atomic resolution convergent beam electron diffraction analysis using convolutional neural networks Zhang, Chenyu; Feng, Jie; DaCosta, Luis Rangel Ultramicroscopy, Vol. 210 https://doi.org/10.1016/j.ultramic.2019.112921	journal	March 2020
Training artificial neural networks for precision orientation and strain mapping using 4D electron diffraction datasets Yuan, Renliang; Zhang, Jiong; He, Lingfeng Ultramicroscopy https://doi.org/10.1016/j.ultramic.2021.113256	journal	March 2021
Contamination of TEM Holders Quantified and Mitigated With the Open-Hardware, High-Vacuum Bakeout System Goh, Yin Min; Schwartz, Jonathan; Rennich, Emily Microscopy and Microanalysis, Vol. 26, Issue 5 https://doi.org/10.1017/S1431927620001762	journal	July 2020
Optimal STEM Convergence Angle Selection Using a Convolutional Neural Network and the Strehl Ratio Schnitzer, Noah; Sung, Suk Hyun; Hovden, Robert Microscopy and Microanalysis, Vol. 26, Issue 5 https://doi.org/10.1017/S1431927620001841	journal	August 2020
Automated Acquisition and Deep Learning of 2D Materials on the Million-Atom Scale Lee, Chia-Hao; Khan, Abid; Zhang, Yue Microscopy and Microanalysis, Vol. 28, Issue S1 https://doi.org/10.1017/S1431927622011424	journal	August 2022
An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics Olszta, Matthew; Hopkins, Derek; Fiedler, Kevin R. Microscopy and Microanalysis https://doi.org/10.1017/S1431927622012065	journal	June 2022
Towards Augmented Microscopy with Reinforcement Learning-Enhanced Workflows Xu, Michael; Kumar, Abinash; LeBeau, James M. Microscopy and Microanalysis, Vol. 28, Issue 6 https://doi.org/10.1017/S1431927622012193	journal	December 2022
Deep Learning Enabled Strain Mapping of Single-Atom Defects in Two-Dimensional Transition Metal Dichalcogenides with Sub-Picometer Precision Lee, Chia-Hao; Khan, Abid; Luo, Di Nano Letters, Vol. 20, Issue 5 https://doi.org/10.1021/acs.nanolett.0c00269	journal	April 2020
Defect Coupling and Sub-Angstrom Structural Distortions in W _{1– x} Mo _x S ₂ Monolayers Azizi, Amin; Wang, Yuanxi; Stone, Greg Nano Letters, Vol. 17, Issue 5 https://doi.org/10.1021/acs.nanolett.6b05045	journal	April 2017
Probing Electron Beam Induced Transformations on a Single-Defect Level via Automated Scanning Transmission Electron Microscopy Roccapriore, Kevin M.; Boebinger, Matthew G.; Dyck, Ondrej ACS Nano, Vol. 16, Issue 10 https://doi.org/10.1021/acsnano.2c07451	journal	October 2022
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem ACS Nano, Vol. 11, Issue 12 https://doi.org/10.1021/acsnano.7b07504	journal	October 2016
Mechanism of Gold-Assisted Exfoliation of Centimeter-Sized Transition-Metal Dichalcogenide Monolayers Velický, Matěj; Donnelly, Gavin E.; Hendren, William R. ACS Nano, Vol. 12, Issue 10 https://doi.org/10.1021/acsnano.8b06101	journal	September 2018
Point Defects and Localized Excitons in 2D WSe ₂ Zheng, Yu Jie; Chen, Yifeng; Huang, Yu Li ACS Nano, Vol. 13, Issue 5 https://doi.org/10.1021/acsnano.9b02316	journal	April 2019
Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy Krivanek, Ondrej L.; Chisholm, Matthew F.; Nicolosi, Valeria Nature, Vol. 464, Issue 7288 https://doi.org/10.1038/nature08879	journal	March 2010
Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2 Maksov, Artem; Dyck, Ondrej; Wang, Kai npj Computational Materials, Vol. 5, Issue 1 https://doi.org/10.1038/s41524-019-0152-9	journal	February 2019
Defect detection in atomic-resolution images via unsupervised learning with translational invariance Guo, Yueming; Kalinin, Sergei V.; Cai, Hui npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00642-1	journal	November 2021
Uncovering material deformations via machine learning combined with four-dimensional scanning transmission electron microscopy Shi, Chuqiao; Cao, Michael C.; Rehn, Sarah M. npj Computational Materials, Vol. 8, Issue 1 https://doi.org/10.1038/s41524-022-00793-9	journal	May 2022
Disentangling multiple scattering with deep learning: application to strain mapping from electron diffraction patterns Munshi, Joydeep; Rakowski, Alexander; Savitzky, Benjamin H. npj Computational Materials, Vol. 8, Issue 1 https://doi.org/10.1038/s41524-022-00939-9	journal	December 2022
Towards data-driven next-generation transmission electron microscopy Spurgeon, Steven R.; Ophus, Colin; Jones, Lewys Nature Materials https://doi.org/10.1038/s41563-020-00833-z	journal	October 2020
TEMImageNet training library and AtomSegNet deep-learning models for high-precision atom segmentation, localization, denoising, and deblurring of atomic-resolution images Lin, Ruoqian; Zhang, Rui; Wang, Chunyang Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-021-84499-w	journal	March 2021
Machine learning in scanning transmission electron microscopy Kalinin, Sergei V.; Ophus, Colin; Voyles, Paul M. Nature Reviews Methods Primers, Vol. 2, Issue 1 https://doi.org/10.1038/s43586-022-00095-w	journal	March 2022
Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook Botifoll, Marc; Pinto-Huguet, Ivan; Arbiol, Jordi Nanoscale Horizons, Vol. 7, Issue 12 https://doi.org/10.1039/D2NH00377E	journal	January 2022
The two-dimensional phase of boron nitride: Few-atomic-layer sheets and suspended membranes Pacilé, D.; Meyer, J. C.; Girit, Ç. Ö. Applied Physics Letters, Vol. 92, Issue 13 https://doi.org/10.1063/1.2903702	journal	March 2008
Quantify point defects in monolayer tungsten diselenide Ding, Sujuan; Lin, Fang; Jin, Chuanhong Nanotechnology, Vol. 32, Issue 25 https://doi.org/10.1088/1361-6528/abeeb2	journal	March 2021
Quantitative ADF STEM: acquisition, analysis and interpretation Jones, L. IOP Conference Series: Materials Science and Engineering, Vol. 109 https://doi.org/10.1088/1757-899X/109/1/012008	journal	February 2016
Deep learning in electron microscopy Ede, Jeffrey M. Machine Learning: Science and Technology, Vol. 2, Issue 1 https://doi.org/10.1088/2632-2153/abd614	journal	March 2021
Intrinsic Instability of Aberration-Corrected Electron Microscopes Schramm, S. M.; van der Molen, S. J.; Tromp, R. M. Physical Review Letters, Vol. 109, Issue 16 https://doi.org/10.1103/PhysRevLett.109.163901	journal	October 2012
Image-to-Image Translation with Conditional Adversarial Networks Isola, Phillip; Zhu, Jun-Yan; Zhou, Tinghui 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.632	conference	July 2017
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks Zhu, Jun-Yan; Park, Taesung; Isola, Phillip 2017 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2017.244	conference	October 2017
Removing Imaging Artifacts in Electron Microscopy using an Asymmetrically Cyclic Adversarial Network without Paired Training Data Quan, Tran Minh; Hildebrand, David Grant Colburn; Lee, Kanggeun 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) https://doi.org/10.1109/ICCVW.2019.00473	conference	October 2019
Deep snow: synthesizing remote sensing imagery with generative adversarial nets Ren, Christopher; Ziemann, Amanda K.; Theiler, James Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imagery XXVI https://doi.org/10.1117/12.2560716	conference	May 2020
Decoding crystallography from high-resolution electron imaging and diffraction datasets with deep learning Aguiar, J. A.; Gong, M. L.; Unocic, R. R. Science Advances, Vol. 5, Issue 10 https://doi.org/10.1126/sciadv.aaw1949	journal	October 2019
Imaging Atomic Rearrangements in Two-Dimensional Silica Glass: Watching Silica's Dance Huang, P. Y.; Kurasch, S.; Alden, J. S. Science, Vol. 342, Issue 6155 https://doi.org/10.1126/science.1242248	journal	October 2013
Crystal symmetry determination in electron diffraction using machine learning Kaufmann, Kevin; Zhu, Chaoyi; Rosengarten, Alexander S. Science, Vol. 367, Issue 6477 https://doi.org/10.1126/science.aay3062	journal	January 2020
Noise2Atom: unsupervised denoising for scanning transmission electron microscopy images Wang, Feng; Henninen, Trond R.; Keller, Debora Applied Microscopy, Vol. 50, Issue 1 https://doi.org/10.1186/s42649-020-00041-8	journal	October 2020
On Information and Sufficiency Kullback, S.; Leibler, R. A. The Annals of Mathematical Statistics, Vol. 22, Issue 1 https://doi.org/10.1214/aoms/1177729694	journal	March 1951

Similar Records

Cycle-Consistent Adversarial Networks for Realistic Pervasive Change Generation in Remote Sensing Imagery

Journal Article · 2020 · Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI) · OSTI ID:1823745

Ren, Christopher X.; Ziemann, Amanda; Theiler, James; +1 more

CosmoGAN: creating high-fidelity weak lensing convergence maps using Generative Adversarial Networks

Journal Article · 2019 · Computational Astrophysics and Cosmology · OSTI ID:1619416

Mustafa, Mustafa; Bard, Deborah; Bhimji, Wahid; +3 more

Fracture Characterization Via AI‐Assisted Analysis of Temperature Logs

Journal Article · 2024 · Journal of Geophysical Research: Machine Learning and Computation · OSTI ID:2437840

Yang, Xiaoyu; Horne, Roland N.; Tartakovsky, Daniel M.

Related Subjects

77 NANOSCIENCE AND NANOTECHNOLOGY
artificial intelligence
computational methods
machine learning
transmission electron microscopy

Title: Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

Citation Formats

References (49)

Similar Records

Related Subjects