An artificial intelligence’s interpretation of complex high-resolution in situ transmission electron microscopy data

Wang, Xingzhi; Yan, Chang; Ondry, Justin C.; Bodiwala, Viraj; Ercius, Peter; Alivisatos, A. Paul

doi:10.1016/j.matt.2023.10.023

An artificial intelligence’s interpretation of complex high-resolution in situ transmission electron microscopy data

Journal Article · Fri Nov 17 04:00:00 EST 2023 · Matter (Online)

DOI:https://doi.org/10.1016/j.matt.2023.10.023· OSTI ID:2329459

^[1]; Yan, Chang ^[2]; Ondry, Justin C. ^[3]; Bodiwala, Viraj ^[4]; Ercius, Peter ^[5]; Alivisatos, A. Paul ^[6]

Univ. of California, Berkeley, CA (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Univ. of California, Berkeley, CA (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Shanghai Jiao Tong Univ. (China)
Univ. of California, Berkeley, CA (United States); Kavli Energy NanoScience Institute, Berkeley, CA (United States); Univ. of Chicago, IL (United States)
Univ. of Chicago, IL (United States)
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). Molecular Foundry
Univ. of California, Berkeley, CA (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Univ. of Chicago, IL (United States)

Complicated nano- and atomic-scale processes with sub-angstrom spatial resolution and millisecond time resolution visualized by in situ transmission electron microscopy (TEM) are often highly dynamical and time consuming to analyze and interpret. Here, we report how variational autoencoders (VAEs) can provide an artificial intelligence’s interpretation of high-resolution in situ TEM data by condensing and deconvoluting complicated atomic-scale dynamics into a latent space with reduced dimensionality. We designed a VAE model with high latent dimensions capable of deconvoluting information from complex high-resolution TEM data. We demonstrate how this model, with high latent dimensions trained on atomically resolved TEM images of lead sulfide (PbS) nanocrystals, is able to capture movements and perturbations of periodic lattices in both simulated and real in situ TEM data. Importantly, the VAE model shows the capability of detecting and deconvoluting dynamical nanoscale physical processes, such as the rotation of crystal lattices and intra-particle ripening during the annealing of semiconductor nanocrystals.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF); USDOE Office of Science (SC), Basic Energy Sciences (BES). Materials Sciences & Engineering Division (MSE)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 2329459

Journal Information:: Matter (Online), Journal Name: Matter (Online) Journal Issue: 1 Vol. 7; ISSN 2590-2385

Publisher:: Cell Press/ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (38)

The Development of the Electron Microscope and of Electron Microscopy(Nobel Lecture) Ruska, Ernst Angewandte Chemie International Edition in English, Vol. 26, Issue 7 https://doi.org/10.1002/anie.198705953	journal	July 1987
The S-state model: a work horse for HRTEM Geuens, P.; Van Dyck, D. Ultramicroscopy, Vol. 93, Issue 3-4 https://doi.org/10.1016/S0304-3991(02)00276-0	journal	December 2002
Do you believe that atoms stay in place when you observe them in HREM? Van Dyck, Dirk; Lobato, Ivan; Chen, Fu-Rong Micron, Vol. 68 https://doi.org/10.1016/j.micron.2014.09.003	journal	January 2015
Neural network approach for ferroelectric hafnium oxide phase identification at the atomistic scale Cheng, Zhiheng; Xie, Xingran; Yang, Yimin Materials Today Electronics, Vol. 3 https://doi.org/10.1016/j.mtelec.2023.100027	journal	May 2023
Current status and future directions for in situ transmission electron microscopy Taheri, Mitra L.; Stach, Eric A.; Arslan, Ilke Ultramicroscopy, Vol. 170 https://doi.org/10.1016/j.ultramic.2016.08.007	journal	November 2016
High-throughput, semi-automated quantitative STEM mass measurement of supported metal nanoparticles using a conventional TEM/STEM House, Stephen D.; Chen, Yuxiang; Jin, Rongchao Ultramicroscopy, Vol. 182 https://doi.org/10.1016/j.ultramic.2017.07.004	journal	November 2017
Multiplexed TEM Specimen Preparation and Analysis of Plasmonic Nanoparticles Mulligan, Sėan K.; Speir, Jeffrey A.; Razinkov, Ivan Microscopy and Microanalysis, Vol. 21, Issue 4 https://doi.org/10.1017/S1431927615014324	journal	July 2015
Deep Learning Computer Vision for Anomaly Detection in Scanning Transmission Electron Microscopy Prifti, Enea; Klie, Robert; Buban, James Microscopy and Microanalysis, Vol. 28, Issue S1 https://doi.org/10.1017/S1431927622011291	journal	August 2022
An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics Olszta, Matthew; Hopkins, Derek; Fiedler, Kevin R. Microscopy and Microanalysis https://doi.org/10.1017/S1431927622012065	journal	June 2022
Understanding the Influence of Receptive Field and Network Complexity in Neural Network-Guided TEM Image Analysis Sytwu, Katherine; Groschner, Catherine; Scott, Mary C. Microscopy and Microanalysis, Vol. 28, Issue 6 https://doi.org/10.1017/S1431927622012466	journal	December 2022
Unraveling Kinetically-Driven Mechanisms of Gold Nanocrystal Shape Transformations Using Graphene Liquid Cell Electron Microscopy Hauwiller, Matthew R.; Frechette, Layne B.; Jones, Matthew R. Nano Letters, Vol. 18, Issue 9 https://doi.org/10.1021/acs.nanolett.8b02337	journal	August 2018
Mapping Defect Relaxation in Quantum Dot Solids upon In Situ Heating Smeaton, Michelle A.; El Baggari, Ismail; Balazs, Daniel M. ACS Nano, Vol. 15, Issue 1 https://doi.org/10.1021/acsnano.0c06990	journal	January 2021
Colloidal Synthesis Path to 2D Crystalline Quantum Dot Superlattices Ondry, Justin C.; Philbin, John P.; Lostica, Michael ACS Nano, Vol. 15, Issue 2 https://doi.org/10.1021/acsnano.0c07202	journal	December 2020
Dynamics and Removal Pathway of Edge Dislocations in Imperfectly Attached PbTe Nanocrystal Pairs: Toward Design Rules for Oriented Attachment Ondry, Justin C.; Hauwiller, Matthew R.; Alivisatos, A. Paul ACS Nano, Vol. 12, Issue 4 https://doi.org/10.1021/acsnano.8b00638	journal	February 2018
Resilient Pathways to Atomic Attachment of Quantum Dot Dimers and Artificial Solids from Faceted CdSe Quantum Dot Building Blocks Ondry, Justin C.; Philbin, John P.; Lostica, Michael ACS Nano, Vol. 13, Issue 11 https://doi.org/10.1021/acsnano.9b03052	journal	June 2019
Setting Carriers Free: Healing Faulty Interfaces Promotes Delocalization and Transport in Nanocrystal Solids Walravens, Willem; Solano, Eduardo; Geenen, Filip ACS Nano, Vol. 13, Issue 11 https://doi.org/10.1021/acsnano.9b04757	journal	November 2019
Neck Barrier Engineering in Quantum Dot Dimer Molecules via Intraparticle Ripening Cui, Jiabin; Koley, Somnath; Panfil, Yossef E. Journal of the American Chemical Society, Vol. 143, Issue 47 https://doi.org/10.1021/jacs.1c08863	journal	November 2021
AutoDetect-mNP: An Unsupervised Machine Learning Algorithm for Automated Analysis of Transmission Electron Microscope Images of Metal Nanoparticles Wang, Xingzhi; Li, Jie; Ha, Hyun Dong JACS Au, Vol. 1, Issue 3 https://doi.org/10.1021/jacsau.0c00030	journal	February 2021
Low-Temperature Nanocrystal Unification through Rotations and Relaxations Probed by in Situ Transmission Electron Microscopy van Huis, Marijn A.; Kunneman, Lucas T.; Overgaag, Karin Nano Letters, Vol. 8, Issue 11 https://doi.org/10.1021/nl8024467	journal	November 2008
Monodisperse, Air-Stable PbS Nanocrystals via Precursor Stoichiometry Control Weidman, Mark C.; Beck, Megan E.; Hoffman, Rachel S. ACS Nano, Vol. 8, Issue 6 https://doi.org/10.1021/nn5018654	journal	May 2014
Computed crystal structure images for high resolution electron microscopy O'Keefe, M. A.; Buseck, P. R.; Iijima, S. Nature, Vol. 274, Issue 5669 https://doi.org/10.1038/274322a0	journal	July 1978
Electron tomography at 2.4-ångström resolution Scott, M. C.; Chen, Chien-Chun; Mecklenburg, Matthew Nature, Vol. 483, Issue 7390 https://doi.org/10.1038/nature10934	journal	March 2012
Colloidal quantum dot molecules manifesting quantum coupling at room temperature Cui, Jiabin; Panfil, Yossef E.; Koley, Somnath Nature Communications, Vol. 10, Issue 1 https://doi.org/10.1038/s41467-019-13349-1	journal	December 2019
Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2 Maksov, Artem; Dyck, Ondrej; Wang, Kai npj Computational Materials, Vol. 5, Issue 1 https://doi.org/10.1038/s41524-019-0152-9	journal	February 2019
Kinetic pathways of crystallization at the nanoscale Ou, Zihao; Wang, Ziwei; Luo, Binbin Nature Materials, Vol. 19, Issue 4 https://doi.org/10.1038/s41563-019-0514-1	journal	October 2019
Towards data-driven next-generation transmission electron microscopy Spurgeon, Steven R.; Ophus, Colin; Jones, Lewys Nature Materials https://doi.org/10.1038/s41563-020-00833-z	journal	October 2020
Software tools for automated transmission electron microscopy Schorb, Martin; Haberbosch, Isabella; Hagen, Wim J. H. Nature Methods, Vol. 16, Issue 6 https://doi.org/10.1038/s41592-019-0396-9	journal	May 2019
DefectTrack: a deep learning-based multi-object tracking algorithm for quantitative defect analysis of in-situ TEM videos in real-time Sainju, Rajat; Chen, Wei-Ying; Schaefer, Samuel Scientific Reports, Vol. 12, Issue 1 https://doi.org/10.1038/s41598-022-19697-1	journal	September 2022
AtomAI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy Ziatdinov, Maxim; Ghosh, Ayana; Wong, Chun Yin Nature Machine Intelligence, Vol. 4, Issue 12 https://doi.org/10.1038/s42256-022-00555-8	journal	December 2022
Sub-10 nm monodisperse PbS cubes by post-synthesis shape engineering Zhang, Haitao; Yang, Jun; Hanrath, Tobias Phys. Chem. Chem. Phys., Vol. 16, Issue 28 https://doi.org/10.1039/C4CP01531B	journal	January 2014
Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook Botifoll, Marc; Pinto-Huguet, Ivan; Arbiol, Jordi Nanoscale Horizons, Vol. 7, Issue 12 https://doi.org/10.1039/D2NH00377E	journal	January 2022
Anomalous nanoparticle surface diffusion in LCTEM is revealed by deep learning-assisted analysis Jamali, Vida; Hargus, Cory; Ben-Moshe, Assaf Proceedings of the National Academy of Sciences, Vol. 118, Issue 10 https://doi.org/10.1073/pnas.2017616118	journal	March 2021
Towards automating structural discovery in scanning transmission electron microscopy ^* Creange, Nicole; Dyck, Ondrej; Vasudevan, Rama K. Machine Learning: Science and Technology, Vol. 3, Issue 1 https://doi.org/10.1088/2632-2153/ac3844	journal	February 2022
Automated data collection in single particle electron microscopy Tan, Yong Zi; Cheng, Anchi; Potter, Clinton S. Microscopy, Vol. 65, Issue 1 https://doi.org/10.1093/jmicro/dfv369	journal	December 2015
Facet-selective etching trajectories of individual semiconductor nanocrystals Yan, Chang; Byrne, Dana; Ondry, Justin C. Science Advances, Vol. 8, Issue 32 https://doi.org/10.1126/sciadv.abq1700	journal	August 2022
High-Resolution EM of Colloidal Nanocrystal Growth Using Graphene Liquid Cells Yuk, Jong Min; Park, Jungwon; Ercius, Peter Science, Vol. 336, Issue 6077 https://doi.org/10.1126/science.1217654	journal	April 2012
Deep learning for electron and scanning probe microscopy: From materials design to atomic fabrication Kalinin, Sergei V.; Ziatdinov, Maxim; Spurgeon, Steven R. MRS Bulletin, Vol. 47, Issue 9 https://doi.org/10.1557/s43577-022-00413-3	journal	September 2022
Theory+AI/ML for microscopy and spectroscopy: Challenges and opportunities Unruh, Davis; Kolluru, Venkata Surya Chaitanya; Baskaran, Arun MRS Bulletin, Vol. 47, Issue 10 https://doi.org/10.1557/s43577-022-00446-8	journal	October 2022

Similar Records

Dimensionality Reduction of SDSS Spectra with Variational Autoencoders

Journal Article · Wed Jul 01 00:00:00 EDT 2020 · The Astronomical Journal (Online) · OSTI ID:23013277

Enhancing Interpretability in Generative Modeling: Statistically Disentangled Latent Spaces Guided by Generative Factors in Scientific Datasets

Journal Article · Fri Aug 01 00:00:00 EDT 2025 · Machine Learning · OSTI ID:2584766

Real-time tracking of structural evolution in 2D MXenes using theory-enhanced machine learning

Journal Article · Fri Aug 02 00:00:00 EDT 2024 · Scientific Reports · OSTI ID:2439101

Related Subjects

47 OTHER INSTRUMENTATION
colloidal nanocrystals
machine learning
transmission electron microscopy

An artificial intelligence’s interpretation of complex high-resolution in situ transmission electron microscopy data

Citation Formats

References (38)

Similar Records

Related Subjects