An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics

Olszta, Matthew; Hopkins, Derek; Fiedler, Kevin R.; Oostrom, Marjolein; Akers, Sarah; Spurgeon, Steven R.

doi:10.1017/S1431927622012065

An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics

Journal Article · Sat Oct 01 04:00:00 EDT 2022 · Microscopy and Microanalysis

DOI:https://doi.org/10.1017/S1431927622012065· OSTI ID:1873446

; Hopkins, Derek; ; ; ;

Abstract

Artificial intelligence (AI) promises to reshape scientific inquiry and enable breakthrough discoveries in areas such as energy storage, quantum computing, and biomedicine. Scanning transmission electron microscopy (STEM), a cornerstone of the study of chemical and materials systems, stands to benefit greatly from AI-driven automation. However, present barriers to low-level instrument control, as well as generalizable and interpretable feature detection, make truly automated microscopy impractical. Here, we discuss the design of a closed-loop instrument control platform guided by emerging sparse data analytics. We hypothesize that a centralized controller, informed by machine learning combining limited a priori knowledge and task-based discrimination, could drive on-the-fly experimental decision-making. This platform may unlock practical, automated analysis of a variety of material features, enabling new high-throughput and statistical studies.

View Journal Article

Sponsoring Organization:: USDOE

OSTI ID:: 1873446

Alternate ID(s):: OSTI ID: 1896593

Journal Information:: Microscopy and Microanalysis, Journal Name: Microscopy and Microanalysis Journal Issue: 5 Vol. 28; ISSN 1431-9276

Publisher:: Oxford University PressCopyright Statement

Country of Publication:: United States

Language:: English

References (61)

Recent Progress of In Situ Transmission Electron Microscopy for Energy Materials Zhang, Chao; Firestein, Konstantin L.; Fernando, Joseph F. S. Advanced Materials, Vol. 32, Issue 18 https://doi.org/10.1002/adma.201904094	journal	September 2019
A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images Madsen, Jacob; Liu, Pei; Kling, Jens Advanced Theory and Simulations, Vol. 1, Issue 8 https://doi.org/10.1002/adts.201800037	journal	July 2018
Leginon: An Automated System for Acquisition of Images from Vitreous Ice Specimens Carragher, Bridget; Kisseberth, Nick; Kriegman, David Journal of Structural Biology, Vol. 132, Issue 1 https://doi.org/10.1006/jsbi.2000.4314	journal	October 2000
The 2017 Nobel Prize in Chemistry: cryo-EM comes of age Shen, Peter S. Analytical and Bioanalytical Chemistry, Vol. 410, Issue 8 https://doi.org/10.1007/s00216-018-0899-8	journal	February 2018
Possibility of an integrated transmission electron microscope: enabling complex in-situ experiments Hattar, Khalid; Jungjohann, Katherine L. Journal of Materials Science, Vol. 56, Issue 9 https://doi.org/10.1007/s10853-020-05598-z	journal	January 2021
AutoEM: a software for automated acquisition and analysis of nanoparticles Uusimaeki, Toni; Wagner, Thorsten; Lipinski, Hans-Gerd Journal of Nanoparticle Research, Vol. 21, Issue 6 https://doi.org/10.1007/s11051-019-4555-9	journal	June 2019
The Materials Data Facility: Data Services to Advance Materials Science Research Blaiszik, B.; Chard, K.; Pruyne, J. JOM, Vol. 68, Issue 8 https://doi.org/10.1007/s11837-016-2001-3	journal	July 2016
Simple method for determining the absolute sense of image rotation in a transmission electron microscope Nakahara, S.; Cullis, A. G. Ultramicroscopy, Vol. 45, Issue 3-4 https://doi.org/10.1016/0304-3991(92)90148-D	journal	November 1992
Combinatorial materials synthesis Takeuchi, Ichiro; Lauterbach, Jochen; Fasolka, Michael J. Materials Today, Vol. 8, Issue 10 https://doi.org/10.1016/S1369-7021(05)71121-4	journal	October 2005
Crystallographic prediction from diffraction and chemistry data for higher throughput classification using machine learning Aguiar, Jeffery A.; Gong, Matthew L.; Tasdizen, Tolga Computational Materials Science, Vol. 173 https://doi.org/10.1016/j.commatsci.2019.109409	journal	February 2020
Design of a graphical user interface for few-shot machine learning classification of electron microscopy data Doty, Christina; Gallagher, Shaun; Cui, Wenqi Computational Materials Science, Vol. 203 https://doi.org/10.1016/j.commatsci.2021.111121	journal	February 2022
Informatics and data science in materials microscopy Voyles, Paul M. Current Opinion in Solid State and Materials Science, Vol. 21, Issue 3 https://doi.org/10.1016/j.cossms.2016.10.001	journal	June 2017
Big data with cognitive computing: A review for the future Gupta, Shivam; Kar, Arpan Kumar; Baabdullah, Abdullah International Journal of Information Management, Vol. 42 https://doi.org/10.1016/j.ijinfomgt.2018.06.005	journal	October 2018
Automated screening of 2D crystallization trials using transmission electron microscopy: A high-throughput tool-chain for sample preparation and microscopic analysis Coudray, Nicolas; Hermann, Gilles; Caujolle-Bert, Daniel Journal of Structural Biology, Vol. 173, Issue 2 https://doi.org/10.1016/j.jsb.2010.09.019	journal	February 2011
Automated crystal orientation and phase mapping in TEM Rauch, E. F.; Véron, M. Materials Characterization, Vol. 98 https://doi.org/10.1016/j.matchar.2014.08.010	journal	December 2014
Autonomous experimentation systems for materials development: A community perspective Stach, Eric; DeCost, Brian; Kusne, A. Gilad Matter, Vol. 4, Issue 9 https://doi.org/10.1016/j.matt.2021.06.036	journal	September 2021
Antimicrobial activity of transition metal acid MoO3 prevents microbial growth on material surfaces Zollfrank, Cordt; Gutbrod, Kai; Wechsler, Peter Materials Science and Engineering: C, Vol. 32, Issue 1 https://doi.org/10.1016/j.msec.2011.09.010	journal	January 2012
Automation of macromolecular crystallography beamlines Arzt, Steffi; Beteva, Antonia; Cipriani, Florent Progress in Biophysics and Molecular Biology, Vol. 89, Issue 2 https://doi.org/10.1016/j.pbiomolbio.2004.09.003	journal	October 2005
Next-Generation Experimentation with Self-Driving Laboratories Häse, Florian; Roch, Loïc M.; Aspuru-Guzik, Alán Trends in Chemistry, Vol. 1, Issue 3 https://doi.org/10.1016/j.trechm.2019.02.007	journal	June 2019
Scripting-customised microscopy tools for Digital Micrograph™ Mitchell, D. R. G.; Schaffer, B. Ultramicroscopy, Vol. 103, Issue 4 https://doi.org/10.1016/j.ultramic.2005.02.003	journal	July 2005
Introducing measure-by-wire, the systematic use of systems and control theory in transmission electron microscopy Tejada, Arturo; den Dekker, Arnold J.; Van den Broek, Wouter Ultramicroscopy, Vol. 111, Issue 11 https://doi.org/10.1016/j.ultramic.2011.08.011	journal	November 2011
Current status and future directions for in situ transmission electron microscopy Taheri, Mitra L.; Stach, Eric A.; Arslan, Ilke Ultramicroscopy, Vol. 170 https://doi.org/10.1016/j.ultramic.2016.08.007	journal	November 2016
High-throughput, semi-automated quantitative STEM mass measurement of supported metal nanoparticles using a conventional TEM/STEM House, Stephen D.; Chen, Yuxiang; Jin, Rongchao Ultramicroscopy, Vol. 182 https://doi.org/10.1016/j.ultramic.2017.07.004	journal	November 2017
SerialEM: A Program for Automated Tilt Series Acquisition on Tecnai Microscopes Using Prediction of Specimen Position Mastronarde, David N. Microscopy and Microanalysis, Vol. 9, Issue S02 https://doi.org/10.1017/S1431927603445911	journal	July 2003
Nion Swift: Open Source Image Processing Software for Instrument Control, Data Acquisition, Organization, Visualization, and Analysis Using Python. Meyer, Chris; Dellby, Niklas; Hachtel, Jordan A. Microscopy and Microanalysis, Vol. 25, Issue S2 https://doi.org/10.1017/S143192761900134X	journal	August 2019
Automating Electron Microscopy through Machine Learning and USETEM Xu, Michael; Kumar, Abinash; LeBeau, James Microscopy and Microanalysis, Vol. 27, Issue S1 https://doi.org/10.1017/S1431927621010394	journal	July 2021
The Blockchain Integrated Automatic Experiment Platform (BiaeP) Xu, Yanheng; Liu, Rulin; Li, Jiagen The Journal of Physical Chemistry Letters, Vol. 11, Issue 23 https://doi.org/10.1021/acs.jpclett.0c02840	journal	November 2020
Atomic-Level Structural Engineering of Graphene on a Mesoscopic Scale Trentino, Alberto; Madsen, Jacob; Mittelberger, Andreas Nano Letters, Vol. 21, Issue 12 https://doi.org/10.1021/acs.nanolett.1c01214	journal	June 2021
Machine Learning in Nanoscience: Big Data at Small Scales Brown, Keith A.; Brittman, Sarah; Maccaferri, Nicolò Nano Letters, Vol. 20, Issue 1 https://doi.org/10.1021/acs.nanolett.9b04090	journal	December 2019
Low Data Drug Discovery with One-Shot Learning Altae-Tran, Han; Ramsundar, Bharath; Pappu, Aneesh S. ACS Central Science, Vol. 3, Issue 4 https://doi.org/10.1021/acscentsci.6b00367	journal	April 2017
Chemical Robotics Enabled Exploration of Stability in Multicomponent Lead Halide Perovskites via Machine Learning Higgins, Kate; Valleti, Sai Mani; Ziatdinov, Maxim ACS Energy Letters, Vol. 5, Issue 11 https://doi.org/10.1021/acsenergylett.0c01749	journal	October 2020
Automated and Autonomous Experiments in Electron and Scanning Probe Microscopy Kalinin, Sergei V.; Ziatdinov, Maxim; Hinkle, Jacob ACS Nano, Vol. 15, Issue 8 https://doi.org/10.1021/acsnano.1c02104	journal	July 2021
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem ACS Nano, Vol. 11, Issue 12 https://doi.org/10.1021/acsnano.7b07504	journal	October 2016
Functional genomic hypothesis generation and experimentation by a robot scientist King, Ross D.; Whelan, Kenneth E.; Jones, Ffion M. Nature, Vol. 427, Issue 6971 https://doi.org/10.1038/nature02236	journal	January 2004
The high-throughput highway to computational materials design Curtarolo, Stefano; Hart, Gus L. W.; Nardelli, Marco Buongiorno Nature Materials, Vol. 12, Issue 3 https://doi.org/10.1038/nmat3568	journal	February 2013
Big–deep–smart data in imaging for guiding materials design Kalinin, Sergei V.; Sumpter, Bobby G.; Archibald, Richard K. Nature Materials, Vol. 14, Issue 10 https://doi.org/10.1038/nmat4395	journal	September 2015
Fiji: an open-source platform for biological-image analysis Schindelin, Johannes; Arganda-Carreras, Ignacio; Frise, Erwin Nature Methods, Vol. 9, Issue 7 https://doi.org/10.1038/nmeth.2019	journal	June 2012
NIH Image to ImageJ: 25 years of image analysis Schneider, Caroline A.; Rasband, Wayne S.; Eliceiri, Kevin W. Nature Methods, Vol. 9, Issue 7 https://doi.org/10.1038/nmeth.2089	journal	June 2012
The development of cryo-EM into a mainstream structural biology technique Nogales, Eva Nature Methods, Vol. 13, Issue 1 https://doi.org/10.1038/nmeth.3694	journal	December 2015
Advances in the field of single-particle cryo-electron microscopy over the last decade Frank, Joachim Nature Protocols, Vol. 12, Issue 2 https://doi.org/10.1038/nprot.2017.004	journal	January 2017
A petascale automated imaging pipeline for mapping neuronal circuits with high-throughput transmission electron microscopy Yin, Wenjing; Brittain, Derrick; Borseth, Jay Nature Communications, Vol. 11, Issue 1 https://doi.org/10.1038/s41467-020-18659-3	journal	October 2020
Rapid and flexible segmentation of electron microscopy data using few-shot machine learning Akers, Sarah; Kautz, Elizabeth; Trevino-Gavito, Andrea npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00652-z	journal	November 2021
Towards data-driven next-generation transmission electron microscopy Spurgeon, Steven R.; Ophus, Colin; Jones, Lewys Nature Materials https://doi.org/10.1038/s41563-020-00833-z	journal	October 2020
Bayesian reaction optimization as a tool for chemical synthesis Shields, Benjamin J.; Stevens, Jason; Li, Jun Nature, Vol. 590, Issue 7844 https://doi.org/10.1038/s41586-021-03213-y	journal	February 2021
Software tools for automated transmission electron microscopy Schorb, Martin; Haberbosch, Isabella; Hagen, Wim J. H. Nature Methods, Vol. 16, Issue 6 https://doi.org/10.1038/s41592-019-0396-9	journal	May 2019
SciPy 1.0: fundamental algorithms for scientific computing in Python Virtanen, Pauli; Gommers, Ralf; Oliphant, Travis E. Nature Methods https://doi.org/10.1038/s41592-019-0686-2	journal	February 2020
Efficient few-shot machine learning for classification of EBSD patterns Kaufmann, Kevin; Lane, Hobson; Liu, Xiao Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-021-87557-5	journal	April 2021
Gaussian processes for autonomous data acquisition at large-scale synchrotron and neutron facilities Noack, Marcus M.; Zwart, Petrus H.; Ushizima, Daniela M. Nature Reviews Physics, Vol. 3, Issue 10 https://doi.org/10.1038/s42254-021-00345-y	journal	July 2021
Fully Mechanically Controlled Automated Electron Microscopic Tomography Liu, Jinxin; Li, Hongchang; Zhang, Lei Scientific Reports, Vol. 6, Issue 1 https://doi.org/10.1038/srep29231	journal	July 2016
Diverse applications of MoO ₃ for high performance organic photovoltaics: fundamentals, processes and optimization strategies Gong, Yongshuai; Dong, Yiman; Zhao, Biao Journal of Materials Chemistry A, Vol. 8, Issue 3 https://doi.org/10.1039/C9TA12005J	journal	January 2020
Commentary: The Materials Project: A materials genome approach to accelerating materials innovation Jain, Anubhav; Ong, Shyue Ping; Hautier, Geoffroy APL Materials, Vol. 1, Issue 1 https://doi.org/10.1063/1.4812323	journal	July 2013
Deep learning in electron microscopy Ede, Jeffrey M. Machine Learning: Science and Technology, Vol. 2, Issue 1 https://doi.org/10.1088/2632-2153/abd614	journal	March 2021
Histopathological Image Analysis: A Review Gurcan, M. N.; Boucheron, L. E.; Can, A. IEEE Reviews in Biomedical Engineering, Vol. 2 https://doi.org/10.1109/RBME.2009.2034865	journal	January 2009
More Is Less: Signal Processing and the Data Deluge Baraniuk, Richard G. Science, Vol. 331, Issue 6018 https://doi.org/10.1126/science.1197448	journal	February 2011
Aberration-corrected Scanning Transmission Electron Microscopy for Atomic-scale Characterization of Semiconductor Devices Van Benthem, Klaus; Pennycook, Stephen J. ECS Transactions, Vol. 11, Issue 3 https://doi.org/10.1149/1.2779062	journal	September 2007
ImageJ2: ImageJ for the next generation of scientific image data Rueden, Curtis T.; Schindelin, Johannes; Hiner, Mark C. BMC Bioinformatics, Vol. 18, Issue 1 https://doi.org/10.1186/s12859-017-1934-z	journal	November 2017
Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets Belianinov, Alex; Vasudevan, Rama; Strelcov, Evgheni Advanced Structural and Chemical Imaging, Vol. 1, Issue 1 https://doi.org/10.1186/s40679-015-0006-6	journal	May 2015
Haves and have nots must find a better way: The case for open scientific hardware Maia Chagas, André PLOS Biology, Vol. 16, Issue 9 https://doi.org/10.1371/journal.pbio.3000014	journal	September 2018
A data ecosystem to support machine learning in materials science Blaiszik, Ben; Ward, Logan; Schwarting, Marcus MRS Communications, Vol. 9, Issue 4 https://doi.org/10.1557/mrc.2019.118	journal	October 2019
Materials science in the artificial intelligence age: high-throughput library generation, machine learning, and a pathway from correlations to the underpinning physics Vasudevan, Rama K.; Choudhary, Kamal; Mehta, Apurva MRS Communications, Vol. 9, Issue 3 https://doi.org/10.1557/mrc.2019.95	journal	July 2019
Image-Based Cardiac Diagnosis With Machine Learning: A Review Martin-Isla, Carlos; Campello, Victor M.; Izquierdo, Cristian Frontiers in Cardiovascular Medicine, Vol. 7 https://doi.org/10.3389/fcvm.2020.00001	journal	January 2020

Similar Records

Automated defect analysis in electron microscopic images

Journal Article · Wed Jul 18 00:00:00 EDT 2018 · npj Computational Materials · OSTI ID:1462882

Autonomous closed-loop mechanistic investigation of molecular electrochemistry via automation

Journal Article · Sat Mar 30 00:00:00 EDT 2024 · Nature Communications · OSTI ID:2331328

An Automated Scanning Transmission Electron Microscope Guided by Sparse Data Analytics

Citation Formats

References (61)

Similar Records

Related Subjects