Building workflows for an interactive human-in-the-loop automated experiment (hAE) in STEM-EELS

Pratiush, Utkarsh; Roccapriore, Kevin M.; Liu, Yongtao; Duscher, Gerd; Ziatdinov, Maxim; Kalinin, Sergei V.

doi:10.1039/d5dd00033e

Building workflows for an interactive human-in-the-loop automated experiment (hAE) in STEM-EELS

Journal Article · Fri Apr 25 00:00:00 EDT 2025 · Digital Discovery

DOI:https://doi.org/10.1039/d5dd00033e· OSTI ID:2563441

^[1]; Roccapriore, Kevin M. ^[2]; Liu, Yongtao ^[2]; Duscher, Gerd ^[1]; Ziatdinov, Maxim ^[3]; Kalinin, Sergei V. ^[4]

Univ. of Tennessee, Knoxville, TN (United States)
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Center for Nanophase Materials Sciences (CNMS)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Univ. of Tennessee, Knoxville, TN (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Exploring the structural, chemical, and physical properties of matter on the nano- and atomic scales has become possible with the recent advances in aberration-corrected electron energy-loss spectroscopy (EELS) in scanning transmission electron microscopy (STEM). However, the current paradigm of STEM-EELS relies on the classical rectangular grid sampling, in which all surface regions are assumed to be of equal a priori interest. However, this is typically not the case for real-world scenarios, where phenomena of interest are concentrated in a small number of spatial locations, such as interfaces, structural and topological defects, and multi-phase inclusions. One of the foundational problems is the discovery of nanometer- or atomic-scale structures having specific signatures in EELS spectra. Herein, we systematically explore the hyperparameters controlling deep kernel learning (DKL) discovery workflows for STEM-EELS and identify the role of the local structural descriptors and acquisition functions in experiment progression. In agreement with the actual experiment, we observe that for certain parameter combinations the experiment path can be trapped in the local minima. We demonstrate the approaches for monitoring the automated experiment in the real and feature space of the system and knowledge acquisition of the DKL model. Based on these, we construct intervention strategies defining the human-in-the-loop automated experiment (hAE). This approach can be further extended to other techniques including 4D STEM and other forms of spectroscopic imaging. The hAE library is available on Github at https://github.com/utkarshp1161/hAE/tree/main/hAE.

View Journal Article

Research Organization:: Pennsylvania State Univ., University Park, PA (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Basic Energy Sciences (BES). Materials Sciences & Engineering Division (MSE); USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF)

Grant/Contract Number:: SC0021118

OSTI ID:: 2563441

Alternate ID(s):: OSTI ID: 3003214

Journal Information:: Digital Discovery, Journal Name: Digital Discovery Journal Issue: 5 Vol. 4; ISSN 2635-098X

Publisher:: Royal Society of ChemistryCopyright Statement

Country of Publication:: United States

Language:: English

References (35)

Physics Discovery in Nanoplasmonic Systems via Autonomous Experiments in Scanning Transmission Electron Microscopy Roccapriore, Kevin M.; Kalinin, Sergei V.; Ziatdinov, Maxim Advanced Science, Vol. 9, Issue 36 https://doi.org/10.1002/advs.202203422	journal	November 2022
Predictability of Localized Plasmonic Responses in Nanoparticle Assemblies Roccapriore, Kevin M.; Ziatdinov, Maxim; Cho, Shin Hum Small, Vol. 17, Issue 21 https://doi.org/10.1002/smll.202100181	journal	April 2021
Sculpting the Plasmonic Responses of Nanoparticles by Directed Electron Beam Irradiation Roccapriore, Kevin M.; Cho, Shin‐Hum; Lupini, Andrew R. Small, Vol. 18, Issue 1 https://doi.org/10.1002/smll.202105099	journal	November 2021
Gaussian Processes in Machine Learning Rasmussen, Carl Edward Advanced Lectures on Machine Learning https://doi.org/10.1007/978-3-540-28650-9_4	book	January 2004
EELS in the STEM: Determination of materials properties on the atomic scale Browning, N. D.; Wallis, D. J.; Nellist, P. D. Micron, Vol. 28, Issue 5 https://doi.org/10.1016/S0968-4328(97)00033-4	journal	October 1997
EELS in the TEM Egerton, R. F.; Malac, M. Journal of Electron Spectroscopy and Related Phenomena, Vol. 143, Issue 2-3 https://doi.org/10.1016/j.elspec.2003.12.009	journal	May 2005
Recent advances in in-situ transmission electron microscopy techniques for heterogeneous catalysis Qu, Jiangshan; Sui, Manling; Li, Rengui iScience, Vol. 26, Issue 7 https://doi.org/10.1016/j.isci.2023.107072	journal	July 2023
STEM-EELS investigation of c-Si/a-AlO interface for solar cell applications Noircler, Guillaume; Lebreton, Fabien; Drahi, Etienne Micron, Vol. 145 https://doi.org/10.1016/j.micron.2021.103032	journal	June 2021
Explainability and human intervention in autonomous scanning probe microscopy Liu, Yongtao; Ziatdinov, Maxim A.; Vasudevan, Rama K. Patterns, Vol. 4, Issue 11 https://doi.org/10.1016/j.patter.2023.100858	journal	November 2023
Mapping chemical and bonding information using multivariate analysis of electron energy-loss spectrum images Bosman, M.; Watanabe, M.; Alexander, D. T. L. Ultramicroscopy, Vol. 106, Issue 11-12, p. 1024-1032 https://doi.org/10.1016/j.ultramic.2006.04.016	journal	October 2006
Exploring Local Crystal Symmetry with Rotationally Invariant Variational Autoencoders Oxley, Mark P.; Kalinin, Sergei V.; Valleti, Mani Microscopy and Microanalysis, Vol. 28, Issue S1 https://doi.org/10.1017/S1431927622011655	journal	August 2022
Exploring Causal Physical Mechanisms via Non-Gaussian Linear Models and Deep Kernel Learning: Applications for Ferroelectric Domain Structures Liu, Yongtao; Ziatdinov, Maxim; Kalinin, Sergei V. ACS Nano, Vol. 16, Issue 1 https://doi.org/10.1021/acsnano.1c09059	journal	December 2021
Atomically localized plasmon enhancement in monolayer graphene Zhou, Wu; Lee, Jaekwang; Nanda, Jagjit Nature Nanotechnology, Vol. 7, Issue 3 https://doi.org/10.1038/nnano.2011.252	journal	January 2012
Mapping surface plasmons on a single metallic nanoparticle Nelayah, Jaysen; Kociak, Mathieu; Stéphan, Odile Nature Physics, Vol. 3, Issue 5 https://doi.org/10.1038/nphys575	journal	April 2007
Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images Horwath, James P.; Zakharov, Dmitri N.; Mégret, Rémi npj Computational Materials, Vol. 6, Issue 1 https://doi.org/10.1038/s41524-020-00363-x	journal	July 2020
Gaussian process analysis of electron energy loss spectroscopy data: multivariate reconstruction and kernel control Kalinin, Sergei V.; Lupini, Andrew R.; Vasudevan, Rama K. npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00611-8	journal	September 2021
Physics and chemistry from parsimonious representations: image analysis via invariant variational autoencoders Valleti, Mani; Ziatdinov, Maxim; Liu, Yongtao npj Computational Materials, Vol. 10, Issue 1 https://doi.org/10.1038/s41524-024-01250-5	journal	August 2024
Publisher Correction: Principal component analysis Greenacre, Michael; Groenen, Patrick J. F.; Hastie, Trevor Nature Reviews Methods Primers, Vol. 3 https://doi.org/10.1038/s43586-023-00209-y	journal	March 2023
(S)TEM-EELS as an advanced characterization technique for lithium-ion batteries Yu, Lei; Li, Matthew; Wen, Jianguo Materials Chemistry Frontiers, Vol. 5, Issue 14 https://doi.org/10.1039/D1QM00275A	journal	January 2021
Spectrally tunable infrared plasmonic F,Sn:In ₂ O ₃ nanocrystal cubes Cho, Shin Hum; Roccapriore, Kevin M.; Dass, Chandriker Kavir The Journal of Chemical Physics, Vol. 152, Issue 1 https://doi.org/10.1063/1.5139050	journal	January 2020
Using electron energy-loss spectroscopy to measure nanoscale electronic and vibrational dynamics in a TEM Kim, Ye-Jin; Palmer, Levi D.; Lee, Wonseok The Journal of Chemical Physics, Vol. 159, Issue 5 https://doi.org/10.1063/5.0147356	journal	August 2023
Discovering invariant spatial features in electron energy loss spectroscopy images on the mesoscopic and atomic levels Roccapriore, Kevin M.; Ziatdinov, Maxim; Lupini, Andrew R. Journal of Applied Physics, Vol. 135, Issue 11 https://doi.org/10.1063/5.0193607	journal	March 2024
Unraveling the impact of initial choices and in-loop interventions on learning dynamics in autonomous scanning probe microscopy Slautin, Boris N.; Liu, Yongtao; Funakubo, Hiroshi Journal of Applied Physics, Vol. 135, Issue 15 https://doi.org/10.1063/5.0198316	journal	April 2024
Realizing smart scanning transmission electron microscopy using high performance computing Pratiush, Utkarsh; Houston, Austin; Kalinin, Sergei V. Review of Scientific Instruments, Vol. 95, Issue 10 https://doi.org/10.1063/5.0225401	journal	October 2024
Combining variational autoencoders and physical bias for improved microscopy data analysis ^∗ Biswas, Arpan; Ziatdinov, Maxim; Kalinin, Sergei V. Machine Learning: Science and Technology, Vol. 4, Issue 4 https://doi.org/10.1088/2632-2153/acf6a9	journal	October 2023
Deep kernel methods learn better: from cards to process optimization Valleti, Mani; Vasudevan, Rama K.; Ziatdinov, Maxim A. Machine Learning: Science and Technology, Vol. 5, Issue 1 https://doi.org/10.1088/2632-2153/ad1a4f	journal	January 2024
AI-enabled Automation of Atomic Manipulation and Characterization in the STEM Roccapriore, Kevin M.; Boebinger, Matthew G.; Klein, Julian Microscopy and Microanalysis, Vol. 29, Issue Supplement_1 https://doi.org/10.1093/micmic/ozad067.702	journal	July 2023
Human-in-the-Loop: The Future of Machine Learning in Automated Electron Microscopy Kalinin, Sergei V.; Liu, Yongtao; Biswas, Arpan Microscopy Today, Vol. 32, Issue 1 https://doi.org/10.1093/mictod/qaad096	journal	February 2024
Spectroscopic Imaging of Single Atoms Within a Bulk Solid Varela, M.; Findlay, S. D.; Lupini, A. R. Physical Review Letters, Vol. 92, Issue 9 https://doi.org/10.1103/PhysRevLett.92.095502	journal	March 2004
Dynamic STEM-EELS for single-atom and defect measurement during electron beam transformations Roccapriore, Kevin M.; Torsi, Riccardo; Robinson, Joshua Science Advances, Vol. 10, Issue 29 https://doi.org/10.1126/sciadv.adn5899	journal	July 2024
Gaussian Processes for Machine Learning Seeger, Matthias International Journal of Neural Systems, Vol. 14, Issue 02 https://doi.org/10.1142/S0129065704001899	journal	April 2004
Image inpainting Bertalmio, Marcelo; Sapiro, Guillermo; Caselles, Vincent Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 https://doi.org/10.1145/344779.344972	conference	January 2000
STEM-EELS imaging of complex oxides and interfaces Varela, Maria; Gazquez, Jaume; Pennycook, Stephen J. MRS Bulletin, Vol. 37, Issue 01, p. 29-35 https://doi.org/10.1557/mrs.2011.330	journal	January 2012
Hyper-Parameter Optimization: A Review of Algorithms and Applications Yu, Tong; Zhu, Hong arXiv https://doi.org/10.48550/arXiv.2003.05689	preprint	January 2020
Active learning in open experimental environments: selecting the right information channel(s) based on predictability in deep kernel learning Ziatdinov, Maxim; Liu, Yongtao; Kalinin, Sergei V. arXiv https://doi.org/10.48550/arXiv.2203.10181	preprint	January 2022

Building workflows for an interactive human-in-the-loop automated experiment (hAE) in STEM-EELS

References (35)

Similar Records

Related Subjects

Building workflows for an interactive human-in-the-loop automated experiment (hAE) in STEM-EELS

Citation Formats

References (35)

Similar Records

Related Subjects