Physics and chemistry from parsimonious representations: image analysis via invariant variational autoencoders

Valleti, Sai Mani Prudhvi; Ziatdinov, Maxim; Liu, Yongtao; Kalinin, Sergei V.

doi:10.1038/s41524-024-01250-5

Physics and chemistry from parsimonious representations: image analysis via invariant variational autoencoders

Journal Article · Wed Aug 14 00:00:00 EDT 2024 · npj Computational Materials

DOI:https://doi.org/10.1038/s41524-024-01250-5· OSTI ID:2439897

Valleti, Sai Mani Prudhvi ^[1]; Ziatdinov, Maxim ^[2]; ^[3]; ^[4]

Univ. of Tennessee, Knoxville, TN (United States)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Center for Nanophase Materials Sciences (CNMS)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States); Univ. of Tennessee, Knoxville, TN (United States)

Electron, optical, and scanning probe microscopy methods are generating ever increasing volume of image data containing information on atomic and mesoscale structures and functionalities. This necessitates the development of the machine learning methods for discovery of physical and chemical phenomena from the data, such as manifestations of symmetry breaking phenomena in electron and scanning tunneling microscopy images, or variability of the nanoparticles. Variational autoencoders (VAEs) are emerging as a powerful paradigm for the unsupervised data analysis, allowing to disentangle the factors of variability and discover optimal parsimonious representation. Here, we summarize recent developments in VAEs, covering the basic principles and intuition behind the VAEs. The invariant VAEs are introduced as an approach to accommodate scale and translation invariances present in imaging data and separate known factors of variations from the ones to be discovered. We further describe the opportunities enabled by the control over VAE architecture, including conditional, semi-supervised, and joint VAEs. Several case studies of VAE applications for toy models and experimental datasets in Scanning Transmission Electron Microscopy are discussed, emphasizing the deep connection between VAE and basic physical principles. Python codes and datasets discussed in this article are available at https://github.com/saimani5/VAE-tutorials and can be used by researchers as an application guide when applying these to their own datasets.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States); University of Washington, Seattle, WA (United States)

Sponsoring Organization:: USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE Office of Science (SC), Basic Energy Sciences (BES); USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF)

Grant/Contract Number:: AC05-00OR22725; AC05-76RL01830; SC0019288

OSTI ID:: 2439897

Alternate ID(s):: OSTI ID: 2482271
OSTI ID: 2479744
OSTI ID: 2479749

Report Number(s):: PNNL-SA-197172

Journal Information:: npj Computational Materials, Journal Name: npj Computational Materials Journal Issue: 1 Vol. 10; ISSN 2057-3960

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United States

Language:: English

References (46)

In Situ Transmission Electron Microscopy for Energy Materials and Devices Fan, Zheng; Zhang, Liqiang; Baumann, Daniel Advanced Materials, Vol. 31, Issue 33 https://doi.org/10.1002/adma.201900608	journal	June 2019
Disentangling Ferroelectric Wall Dynamics and Identification of Pinning Mechanisms via Deep Learning Liu, Yongtao; Proksch, Roger; Wong, Chun Yin Advanced Materials, Vol. 33, Issue 43 https://doi.org/10.1002/adma.202103680	journal	September 2021
Exploring Physics of Ferroelectric Domain Walls in Real Time: Deep Learning Enabled Scanning Probe Microscopy Liu, Yongtao; Kelley, Kyle P.; Funakubo, Hiroshi Advanced Science, Vol. 9, Issue 31 https://doi.org/10.1002/advs.202203957	journal	September 2022
Origin of Anomalous Pt-Pt Distances in the Pt/Alumina Catalytic System Sohlberg, Karl; Rashkeev, Sergey; Borisevich, Albina Y. ChemPhysChem, Vol. 5, Issue 12 https://doi.org/10.1002/cphc.200400212	journal	December 2004
Building Structures Atom by Atom via Electron Beam Manipulation Dyck, Ondrej; Kim, Songkil; Jimenez-Izal, Elisa Small, Vol. 14, Issue 38 https://doi.org/10.1002/smll.201801771	journal	August 2018
Predictability of Localized Plasmonic Responses in Nanoparticle Assemblies Roccapriore, Kevin M.; Ziatdinov, Maxim; Cho, Shin Hum Small, Vol. 17, Issue 21 https://doi.org/10.1002/smll.202100181	journal	April 2021
Decoding the Mechanisms of Phase Transitions from In Situ Microscopy Observations Valleti, Sai Mani Prudhvi; Ignatans, Reinis; Kalinin, Sergei V. Small, Vol. 18, Issue 40 https://doi.org/10.1002/smll.202104318	journal	September 2022
Multilayer Perceptron (MLP) Taud, H.; Mas, J. F. Geomatic Approaches for Modeling Land Change Scenarios https://doi.org/10.1007/978-3-319-60801-3_27	book	October 2017
E-beam manipulation of Si atoms on graphene edges with an aberration-corrected scanning transmission electron microscope Dyck, Ondrej; Kim, Songkil; Kalinin, Sergei V. Nano Research, Vol. 11, Issue 12 https://doi.org/10.1007/s12274-018-2141-6	journal	July 2018
Current imaging tunneling spectroscopy of metallic deposits on silicon Asenjo, A.; Gómez-Rodríguez, J. M.; Baró, A. M. Ultramicroscopy, Vol. 42-44 https://doi.org/10.1016/0304-3991(92)90381-S	journal	July 1992
Force spectroscopy with single bio-molecules Clausen-Schaumann, Hauke; Seitz, Markus; Krautbauer, Rupert Current Opinion in Chemical Biology, Vol. 4, Issue 5 https://doi.org/10.1016/S1367-5931(00)00126-5	journal	October 2000
Scanning probe microscopy for electrocatalysis Wang, Yuqing; Skaanvik, Sebastian Amland; Xiong, Xuya Matter, Vol. 4, Issue 11 https://doi.org/10.1016/j.matt.2021.09.024	journal	November 2021
Toward Decoding the Relationship between Domain Structure and Functionality in Ferroelectrics via Hidden Latent Variables Kalinin, Sergei V.; Kelley, Kyle; Vasudevan, Rama K. ACS Applied Materials & Interfaces, Vol. 13, Issue 1 https://doi.org/10.1021/acsami.0c15085	journal	January 2021
Exploring Causal Physical Mechanisms via Non-Gaussian Linear Models and Deep Kernel Learning: Applications for Ferroelectric Domain Structures Liu, Yongtao; Ziatdinov, Maxim; Kalinin, Sergei V. ACS Nano, Vol. 16, Issue 1 https://doi.org/10.1021/acsnano.1c09059	journal	December 2021
Disentangling Electronic Transport and Hysteresis at Individual Grain Boundaries in Hybrid Perovskites via Automated Scanning Probe Microscopy Liu, Yongtao; Yang, Jonghee; Lawrie, Benjamin J. ACS Nano, Vol. 17, Issue 10 https://doi.org/10.1021/acsnano.3c03363	journal	May 2023
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem ACS Nano, Vol. 11, Issue 12 https://doi.org/10.1021/acsnano.7b07504	journal	October 2016
SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules Weininger, David Journal of Chemical Information and Modeling, Vol. 28, Issue 1 https://doi.org/10.1021/ci00057a005	journal	February 1988
Going deeper than microscopy: the optical imaging frontier in biology Ntziachristos, Vasilis Nature Methods, Vol. 7, Issue 8 https://doi.org/10.1038/nmeth.1483	journal	July 2010
How the doors to the nanoworld were opened Gerber, Christoph; Lang, Hans Peter Nature Nanotechnology, Vol. 1, Issue 1 https://doi.org/10.1038/nnano.2006.70	journal	October 2006
Scanning probe microscopy for advanced nanoelectronics Hui, Fei; Lanza, Mario Nature Electronics, Vol. 2, Issue 6 https://doi.org/10.1038/s41928-019-0264-8	journal	June 2019
AtomAI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy Ziatdinov, Maxim; Ghosh, Ayana; Wong, Chun Yin Nature Machine Intelligence, Vol. 4, Issue 12 https://doi.org/10.1038/s42256-022-00555-8	journal	December 2022
Magnetic dissipation force microscopy Grütter, P.; Liu, Y.; LeBlanc, P. Applied Physics Letters, Vol. 71, Issue 2 https://doi.org/10.1063/1.119519	journal	July 1997
The influence of atomic structure on the formation of electrical barriers at grain boundaries in SrTiO3 Browning, N. D.; Buban, J. P.; Moltaji, H. O. Applied Physics Letters, Vol. 74, Issue 18 https://doi.org/10.1063/1.123922	journal	May 1999
Placing single atoms in graphene with a scanning transmission electron microscope Dyck, Ondrej; Kim, Songkil; Kalinin, Sergei V. Applied Physics Letters, Vol. 111, Issue 11 https://doi.org/10.1063/1.4998599	journal	September 2017
Magnetic imaging by ‘‘force microscopy’’ with 1000 Å resolution Martin, Y.; Wickramasinghe, H. K. Applied Physics Letters, Vol. 50, Issue 20 https://doi.org/10.1063/1.97800	journal	May 1987
Disentangling ferroelectric domain wall geometries and pathways in dynamic piezoresponse force microscopy via unsupervised machine learning Kalinin, Sergei V.; Steffes, James J.; Liu, Yongtao Nanotechnology, Vol. 33, Issue 5 https://doi.org/10.1088/1361-6528/ac2f5b	journal	November 2021
Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation Krenn, Mario; Häse, Florian; Nigam, AkshatKumar Machine Learning: Science and Technology, Vol. 1, Issue 4 https://doi.org/10.1088/2632-2153/aba947	journal	November 2020
Decoding the shift-invariant data: applications for band-excitation scanning probe microscopy ^* Liu, Yongtao; Vasudevan, Rama K.; Kelley, Kyle K. Machine Learning: Science and Technology, Vol. 2, Issue 4 https://doi.org/10.1088/2632-2153/ac28de	journal	October 2021
Optimizing training trajectories in variational autoencoders via latent Bayesian optimization approach ^* Biswas, Arpan; Vasudevan, Rama; Ziatdinov, Maxim Machine Learning: Science and Technology, Vol. 4, Issue 1 https://doi.org/10.1088/2632-2153/acb316	journal	February 2023
Atomic-resolution spectroscopic imaging: past, present and future Pennycook, S. J.; Varela, M.; Lupini, A. R. Journal of Electron Microscopy, Vol. 58, Issue 3 https://doi.org/10.1093/jmicro/dfn030	journal	January 2009
Atomic-Scale Compensation Phenomena at Polar Interfaces Chisholm, Matthew F.; Luo, Weidong; Oxley, Mark P. Physical Review Letters, Vol. 105, Issue 19 https://doi.org/10.1103/PhysRevLett.105.197602	journal	November 2010
Electronic Structure of the Si(111)2 × 1 Surface by Scanning-Tunneling Microscopy Stroscio, Joseph A.; Feenstra, R. M.; Fein, A. P. Physical Review Letters, Vol. 57, Issue 20 https://doi.org/10.1103/PhysRevLett.57.2579	journal	November 1986
Nonstoichiometry and the Electrical Activity of Grain Boundaries in SrTiO 3 Kim, Miyoung; Duscher, Gerd; Browning, Nigel D. Physical Review Letters, Vol. 86, Issue 18 https://doi.org/10.1103/PhysRevLett.86.4056	journal	April 2001
Gradient-based learning applied to document recognition Lecun, Y.; Bottou, L.; Bengio, Y. Proceedings of the IEEE, Vol. 86, Issue 11 https://doi.org/10.1109/5.726791	journal	January 1998
The microscopes of Antoni van Leeuwenhoek van Zuylen, J. Journal of Microscopy, Vol. 121, Issue 3 https://doi.org/10.1111/j.1365-2818.1981.tb01227.x	journal	March 1981
Electromechanical Imaging and Spectroscopy of Ferroelectric and Piezoelectric Materials: State of the Art and Prospects for the Future Balke, Nina; Bdikin, Igor; Kalinin, Sergei V. Journal of the American Ceramic Society, Vol. 92, Issue 8 https://doi.org/10.1111/j.1551-2916.2009.03240.x	journal	August 2009
Building and exploring libraries of atomic defects in graphene: Scanning transmission electron and scanning tunneling microscopy study Ziatdinov, Maxim; Dyck, Ondrej; Li, Xin Science Advances, Vol. 5, Issue 9 https://doi.org/10.1126/sciadv.aaw8989	journal	September 2019
Exploring order parameters and dynamic processes in disordered systems via variational autoencoders Kalinin, Sergei V.; Dyck, Ondrej; Jesse, Stephen Science Advances, Vol. 7, Issue 17 https://doi.org/10.1126/sciadv.abd5084	journal	April 2021
Single Molecule Force Spectroscopy on Polysaccharides by Atomic Force Microscopy Rief, M. Science, Vol. 275, Issue 5304 https://doi.org/10.1126/science.275.5304.1295	journal	February 1997
ImageNet classification with deep convolutional neural networks Krizhevsky, Alex; Sutskever, Ilya; Hinton, Geoffrey E. Communications of the ACM, Vol. 60, Issue 6 https://doi.org/10.1145/3065386	journal	May 2017
Time-Resolved X-Ray Microscopy for Materials Science Wen, Haidan; Cherukara, Mathew J.; Holt, Martin V. Annual Review of Materials Research, Vol. 49, Issue 1 https://doi.org/10.1146/annurev-matsci-070616-124014	journal	July 2019
Chemical Force Microscopy Noy, Aleksandr; Vezenov, Dmitri V.; Lieber, Charles M. Annual Review of Materials Science, Vol. 27, Issue 1 https://doi.org/10.1146/annurev.matsci.27.1.381	journal	August 1997
Imaging and Control of Domain Structures in Ferroelectric thin Films via Scanning Force Microscopy Gruverman, Alexei; Auciello, Orlando; Tokumoto, Hiroshi Annual Review of Materials Science, Vol. 28, Issue 1 https://doi.org/10.1146/annurev.matsci.28.1.101	journal	August 1998
An Introduction to Variational Autoencoders Kingma, Diederik P.; Welling, Max Foundations and Trends® in Machine Learning, Vol. 12, Issue 4 https://doi.org/10.1561/2200000056	journal	January 2019
Popular Ensemble Methods: An Empirical Study Opitz, D.; Maclin, R. Journal of Artificial Intelligence Research, Vol. 11 https://doi.org/10.1613/jair.614	journal	July 1999
scikit-image: image processing in Python van der Walt, Stéfan; Schönberger, Johannes L.; Nunez-Iglesias, Juan PeerJ, Vol. 2 https://doi.org/10.7717/peerj.453	journal	January 2014

Similar Records

Related Subjects

36 MATERIALS SCIENCE
graphene
materials science

Physics and chemistry from parsimonious representations: image analysis via invariant variational autoencoders

Citation Formats

References (46)

Similar Records

Related Subjects