Towards automating structural discovery in scanning transmission electron microscopy *
Abstract
Abstract Scanning transmission electron microscopy is now the primary tool for exploring functional materials on the atomic level. Often, features of interest are highly localized in specific regions in the material, such as ferroelectric domain walls, extended defects, or second phase inclusions. Selecting regions to image for structural and chemical discovery via atomically resolved imaging has traditionally proceeded via human operators making semi-informed judgements on sampling locations and parameters. Recent efforts at automation for structural and physical discovery have pointed towards the use of ‘active learning’ methods that utilize Bayesian optimization with surrogate models to quickly find relevant regions of interest. Yet despite the potential importance of this direction, there is a general lack of certainty in selecting relevant control algorithms and how to balance a priori knowledge of the material system with knowledge derived during experimentation. Here we address this gap by developing the automated experiment workflows with several combinations to both illustrate the effects of these choices and demonstrate the tradeoffs associated with each in terms of accuracy, robustness, and susceptibility to hyperparameters for structural discovery. We discuss possible methods to build descriptors using the raw image data and deep learning based semantic segmentation, as well as themore »
- Publication Date:
- Research Org.:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE; USDOE Office of Science (SC), Basic Energy Sciences (BES). Materials Sciences & Engineering Division
- OSTI Identifier:
- 1844793
- Alternate Identifier(s):
- OSTI ID: 1829742; OSTI ID: 1860592
- Grant/Contract Number:
- AC05-00OR22725
- Resource Type:
- Published Article
- Journal Name:
- Machine Learning: Science and Technology
- Additional Journal Information:
- Journal Name: Machine Learning: Science and Technology Journal Volume: 3 Journal Issue: 1; Journal ID: ISSN 2632-2153
- Publisher:
- IOP Publishing
- Country of Publication:
- United Kingdom
- Language:
- English
- Subject:
- 47 OTHER INSTRUMENTATION; STEM; Bayesian optimization; automated experiments; data acquisition workflows
Citation Formats
Creange, Nicole, Dyck, Ondrej, Vasudevan, Rama K., Ziatdinov, Maxim, and Kalinin, Sergei V. Towards automating structural discovery in scanning transmission electron microscopy *. United Kingdom: N. p., 2022.
Web. doi:10.1088/2632-2153/ac3844.
Creange, Nicole, Dyck, Ondrej, Vasudevan, Rama K., Ziatdinov, Maxim, & Kalinin, Sergei V. Towards automating structural discovery in scanning transmission electron microscopy *. United Kingdom. https://doi.org/10.1088/2632-2153/ac3844
Creange, Nicole, Dyck, Ondrej, Vasudevan, Rama K., Ziatdinov, Maxim, and Kalinin, Sergei V. Fri .
"Towards automating structural discovery in scanning transmission electron microscopy *". United Kingdom. https://doi.org/10.1088/2632-2153/ac3844.
@article{osti_1844793,
title = {Towards automating structural discovery in scanning transmission electron microscopy *},
author = {Creange, Nicole and Dyck, Ondrej and Vasudevan, Rama K. and Ziatdinov, Maxim and Kalinin, Sergei V.},
abstractNote = {Abstract Scanning transmission electron microscopy is now the primary tool for exploring functional materials on the atomic level. Often, features of interest are highly localized in specific regions in the material, such as ferroelectric domain walls, extended defects, or second phase inclusions. Selecting regions to image for structural and chemical discovery via atomically resolved imaging has traditionally proceeded via human operators making semi-informed judgements on sampling locations and parameters. Recent efforts at automation for structural and physical discovery have pointed towards the use of ‘active learning’ methods that utilize Bayesian optimization with surrogate models to quickly find relevant regions of interest. Yet despite the potential importance of this direction, there is a general lack of certainty in selecting relevant control algorithms and how to balance a priori knowledge of the material system with knowledge derived during experimentation. Here we address this gap by developing the automated experiment workflows with several combinations to both illustrate the effects of these choices and demonstrate the tradeoffs associated with each in terms of accuracy, robustness, and susceptibility to hyperparameters for structural discovery. We discuss possible methods to build descriptors using the raw image data and deep learning based semantic segmentation, as well as the implementation of variational autoencoder based representation. Furthermore, each workflow is applied to a range of feature sizes including NiO pillars within a La:SrMnO 3 matrix, ferroelectric domains in BiFeO 3 , and topological defects in graphene. The code developed in this manuscript is open sourced and will be released at github.com/nccreang/AE_Workflows .},
doi = {10.1088/2632-2153/ac3844},
journal = {Machine Learning: Science and Technology},
number = 1,
volume = 3,
place = {United Kingdom},
year = {Fri Feb 11 00:00:00 EST 2022},
month = {Fri Feb 11 00:00:00 EST 2022}
}
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1088/2632-2153/ac3844
https://doi.org/10.1088/2632-2153/ac3844
Other availability
Search WorldCat to find libraries that may hold this journal
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
The potential for Bayesian compressive sensing to significantly reduce electron dose in high-resolution STEM images
journal, October 2013
- Stevens, Andrew; Yang, Hao; Carin, Lawrence
- Microscopy, Vol. 63, Issue 1
Building Structures Atom by Atom via Electron Beam Manipulation
journal, August 2018
- Dyck, Ondrej; Kim, Songkil; Jimenez-Izal, Elisa
- Small, Vol. 14, Issue 38
Nion Swift: Open Source Image Processing Software for Instrument Control, Data Acquisition, Organization, Visualization, and Analysis Using Python.
journal, August 2019
- Meyer, Chris; Dellby, Niklas; Hachtel, Jordan A.
- Microscopy and Microanalysis, Vol. 25, Issue S2
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations
journal, October 2016
- Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem
- ACS Nano, Vol. 11, Issue 12
Big, Deep, and Smart Data in Scanning Probe Microscopy
journal, September 2016
- Kalinin, Sergei V.; Strelcov, Evgheni; Belianinov, Alex
- ACS Nano, Vol. 10, Issue 10
Atomically Precise Control of Carbon Insertion into hBN Monolayer Point Vacancies using a Focused Electron Beam Guide
journal, May 2021
- Park, Hyoju; Wen, Yi; Li, Sylvia Xin
- Small, Vol. 17, Issue 23
Engineering single-atom dynamics with electron irradiation
journal, May 2019
- Su, Cong; Tripathi, Mukesh; Yan, Qing-Bo
- Science Advances, Vol. 5, Issue 5
Direct Observation of Atomic Dynamics and Silicon Doping at a Topological Defect in Graphene
journal, June 2014
- Yang, Zhiqing; Yin, Lichang; Lee, Jaekwang
- Angewandte Chemie International Edition, Vol. 53, Issue 34
Manipulating low-dimensional materials down to the level of single atoms with electron irradiation
journal, September 2017
- Susi, Toma; Meyer, Jannik C.; Kotakoski, Jani
- Ultramicroscopy, Vol. 180
Unusual electrical conductivity driven by localized stoichiometry modification at vertical epitaxial interfaces
journal, January 2020
- Zhang, Wenrui; Cheng, Shaobo; Rouleau, Christopher M.
- Materials Horizons, Vol. 7, Issue 12
Fire up the atom forge
journal, November 2016
- Kalinin, Sergei V.; Borisevich, Albina; Jesse, Stephen
- Nature, Vol. 539, Issue 7630
Flexible metallic nanowires with self-adaptive contacts to semiconducting transition-metal dichalcogenide monolayers
journal, April 2014
- Lin, Junhao; Cretu, Ovidiu; Zhou, Wu
- Nature Nanotechnology, Vol. 9, Issue 6
Electron-beam introduction of heteroatomic Pt–Si structures in graphene
journal, May 2020
- Dyck, Ondrej; Zhang, Cheng; Rack, Philip D.
- Carbon, Vol. 161
Adaptive probe trajectory scanning probe microscopy for multiresolution measurements of interface geometry
journal, June 2009
- Ovchinnikov, Oleg S.; Jesse, S.; Kalinin, S. V.
- Nanotechnology, Vol. 20, Issue 25
Electron-Beam Manipulation of Silicon Dopants in Graphene
journal, June 2018
- Tripathi, Mukesh; Mittelberger, Andreas; Pike, Nicholas A.
- Nano Letters, Vol. 18, Issue 8
Investigating phase transitions from local crystallographic analysis based on statistical learning of atomic environments in 2D MoS 2 -ReS 2
journal, March 2021
- Vasudevan, Rama K.; Ziatdinov, Maxim; Sharma, Vinit
- Applied Physics Reviews, Vol. 8, Issue 1
Patterned probes for high precision 4D-STEM bragg measurements
journal, February 2020
- Zeltmann, Steven E.; Müller, Alexander; Bustillo, Karen C.
- Ultramicroscopy, Vol. 209
Exploring physics of ferroelectric domain walls via Bayesian analysis of atomically resolved STEM data
journal, December 2020
- Nelson, Christopher T.; Vasudevan, Rama K.; Zhang, Xiaohang
- Nature Communications, Vol. 11, Issue 1
A self-driving microscope and the Atomic Forge
journal, September 2019
- Dyck, Ondrej; Jesse, Stephen; Kalinin, Sergei V.
- MRS Bulletin, Vol. 44, Issue 09
Algorithms and Software for Nanomanipulation with Atomic Force Microscopes
journal, April 2009
- Requicha, A. A. G.; Arbuckle, D. J.; Mokaberi, B.
- The International Journal of Robotics Research, Vol. 28, Issue 4
Applying compressive sensing to TEM video: a substantial frame rate increase on any camera
journal, August 2015
- Stevens, Andrew; Kovarik, Libor; Abellan, Patricia
- Advanced Structural and Chemical Imaging, Vol. 1, Issue 10
Fast Scanning Probe Microscopy via Machine Learning: Non‐Rectangular Scans with Compressed Sensing and Gaussian Process Optimization
journal, August 2020
- Kelley, Kyle P.; Ziatdinov, Maxim; Collins, Liam
- Small, Vol. 16, Issue 37
Disentangling nanoscale electric and magnetic fields by time-reversal operation in differential phase-contrast STEM
journal, October 2020
- Campanini, M.; Nasi, L.; Albertini, F.
- Applied Physics Letters, Vol. 117, Issue 15
Direct Measurement of Surface Diffusion Using Atom-Tracking Scanning Tunneling Microscopy
journal, January 1996
- Swartzentruber, B. S.
- Physical Review Letters, Vol. 76, Issue 3
Open source scanning probe microscopy control software package GXSM
journal, May 2010
- Zahl, Percy; Wagner, Thorsten; Möller, Rolf
- Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena, Vol. 28, Issue 3
Atom-by-atom structural and chemical analysis by annular dark-field electron microscopy
journal, March 2010
- Krivanek, Ondrej L.; Chisholm, Matthew F.; Nicolosi, Valeria
- Nature, Vol. 464, Issue 7288
Direct Observation of Continuous Electric Dipole Rotation in Flux-Closure Domains in Ferroelectric Pb(Zr,Ti)O3
journal, March 2011
- Jia, C. -L.; Urban, K. W.; Alexe, M.
- Science, Vol. 331, Issue 6023
Placing single atoms in graphene with a scanning transmission electron microscope
journal, September 2017
- Dyck, Ondrej; Kim, Songkil; Kalinin, Sergei V.
- Applied Physics Letters, Vol. 111, Issue 11
Direct atomic fabrication and dopant positioning in Si using electron beams with active real-time image-based feedback
journal, April 2018
- Jesse, Stephen; Hudak, Bethany M.; Zarkadoula, Eva
- Nanotechnology, Vol. 29, Issue 25
Electron-Beam Shaping in the Transmission Electron Microscope: Control of Electron-Beam Propagation Along Atomic Columns
journal, April 2019
- Rotunno, E.; Tavabi, A. H.; Yucelen, E.
- Physical Review Applied, Vol. 11, Issue 4
Predictability as a probe of manifest and latent physics: The case of atomic scale structural, chemical, and polarization behaviors in multiferroic Sm-doped BiFeO 3
journal, March 2021
- Ziatdinov, Maxim; Creange, Nicole; Zhang, Xiaohang
- Applied Physics Reviews, Vol. 8, Issue 1
Doping transition-metal atoms in graphene for atomic-scale tailoring of electronic, magnetic, and quantum topological properties
journal, March 2021
- Dyck, Ondrej; Zhang, Lizhi; Yoon, Mina
- Carbon, Vol. 173
Atomic-resolution spectroscopic imaging: past, present and future
journal, January 2009
- Pennycook, S. J.; Varela, M.; Lupini, A. R.
- Journal of Electron Microscopy, Vol. 58, Issue 3
Single atom visibility in STEM optical depth sectioning
journal, October 2016
- Ishikawa, Ryo; Pennycook, Stephen J.; Lupini, Andrew R.
- Applied Physics Letters, Vol. 109, Issue 16
Picometre-precision analysis of scanning transmission electron microscopy images of platinum nanocatalysts
journal, June 2014
- Yankovich, Andrew B.; Berkels, Benjamin; Dahmen, W.
- Nature Communications, Vol. 5, Issue 1
N-FINDR: an algorithm for fast autonomous spectral end-member determination in hyperspectral data
conference, October 1999
- Winter, Michael E.
- SPIE's International Symposium on Optical Science, Engineering, and Instrumentation, SPIE Proceedings
AtomAI: Open-source software for applications of deep learning to microscopy data
journal, July 2021
- Ziatdinov, Maxim; Kalinin, Sergei
- Microscopy and Microanalysis, Vol. 27, Issue S1
Doping of Cr in Graphene Using Electron Beam Manipulation for Functional Defect Engineering
journal, October 2020
- Dyck, Ondrej; Yoon, Mina; Zhang, Lizhi
- ACS Applied Nano Materials, Vol. 3, Issue 11
Silicon Substitution in Nanotubes and Graphene via Intermittent Vacancies
journal, May 2019
- Inani, Heena; Mustonen, Kimmo; Markevich, Alexander
- The Journal of Physical Chemistry C
Depth sectioning with the aberration-corrected scanning transmission electron microscope
journal, February 2006
- Borisevich, A. Y.; Lupini, A. R.; Pennycook, S. J.
- Proceedings of the National Academy of Sciences, Vol. 103, Issue 9
Big data in reciprocal space: Sliding fast Fourier transforms for determining periodicity
journal, March 2015
- Vasudevan, Rama K.; Belianinov, Alex; Gianfrancesco, Anthony G.
- Applied Physics Letters, Vol. 106, Issue 9
Direct Observation of Dopant Atom Diffusion in a Bulk Semiconductor Crystal Enhanced by a Large Size Mismatch
journal, October 2014
- Ishikawa, Ryo; Mishra, Rohan; Lupini, Andrew R.
- Physical Review Letters, Vol. 113, Issue 15
Electron ptychography of 2D materials to deep sub-ångström resolution
journal, July 2018
- Jiang, Yi; Chen, Zhen; Han, Yimo
- Nature, Vol. 559, Issue 7714
Atomic-resolution differential phase contrast STEM on ferroelectric materials: A mean-field approach
journal, May 2020
- Campanini, M.; Eimre, K.; Bon, M.
- Physical Review B, Vol. 101, Issue 18
Partial Scanning Transmission Electron Microscopy with Deep Learning
journal, May 2020
- Ede, Jeffrey M.; Beanland, Richard
- Scientific Reports, Vol. 10, Issue 1
Spectroscopic Imaging of Single Atoms Within a Bulk Solid
journal, March 2004
- Varela, M.; Findlay, S. D.; Lupini, A. R.
- Physical Review Letters, Vol. 92, Issue 9
Generation of Nondiffracting Electron Bessel Beams
journal, January 2014
- Grillo, Vincenzo; Karimi, Ebrahim; Gazzadi, Gian Carlo
- Physical Review X, Vol. 4, Issue 1
Ensemble learning-iterative training machine learning for uncertainty quantification and automated experiment in atom-resolved microscopy
journal, July 2021
- Ghosh, Ayana; Sumpter, Bobby G.; Dyck, Ondrej
- npj Computational Materials, Vol. 7, Issue 1
Dynamic scan control in STEM: spiral scans
journal, June 2016
- Sang, Xiahan; Lupini, Andrew R.; Unocic, Raymond R.
- Advanced Structural and Chemical Imaging, Vol. 2, Issue 1
Temperature Measurement by a Nanoscale Electron Probe Using Energy Gain and Loss Spectroscopy
journal, March 2018
- Idrobo, Juan Carlos; Lupini, Andrew R.; Feng, Tianli
- Physical Review Letters, Vol. 120, Issue 9
Artificial-intelligence-driven scanning probe microscopy
journal, March 2020
- Krull, A.; Hirsch, P.; Rother, C.
- Communications Physics, Vol. 3, Issue 1