Demonstration of an AI-driven workflow for autonomous high-resolution scanning microscopy

Kandel, Saugat; Zhou, Tao; Babu, Anakha V.; Di, Zichao; Li, Xinxin; Ma, Xuedan; Holt, Martin; Miceli, Antonino; Phatak, Charudatta; Cherukara, Mathew J.

doi:10.1038/s41467-023-40339-1

Demonstration of an AI-driven workflow for autonomous high-resolution scanning microscopy

Journal Article · Thu Sep 07 00:00:00 EDT 2023 · Nature Communications

DOI:https://doi.org/10.1038/s41467-023-40339-1· OSTI ID:1998803

; Zhou, Tao; Babu, Anakha V.; Di, Zichao; ; ; ; ; ;

Abstract

Modern scanning microscopes can image materials with up to sub-atomic spatial and sub-picosecond time resolutions, but these capabilities come with large volumes of data, which can be difficult to store and analyze. We report the Fast Autonomous Scanning Toolkit (FAST) that addresses this challenge by combining a neural network, route optimization, and efficient hardware controls to enable a self-driving experiment that actively identifies and measures a sparse but representative data subset in lieu of the full dataset. FAST requires no prior information about the sample, is computationally efficient, and uses generic hardware controls with minimal experiment-specific wrapping. We test FAST in simulations and a dark-field X-ray microscopy experiment of a WSe ₂ film. Our studies show that a FAST scan of <25% is sufficient to accurately image and analyze the sample. FAST is easy to adapt for any scanning microscope; its broad adoption will empower general multi-level studies of materials evolution with respect to time, temperature, or other parameters.

View Journal Article

Sponsoring Organization:: USDOE

Grant/Contract Number:: NONE; AC02-06CH11357

OSTI ID:: 1998803

Alternate ID(s):: OSTI ID: 2409387

Journal Information:: Nature Communications, Journal Name: Nature Communications Journal Issue: 1 Vol. 14; ISSN 2041-1723

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (34)

Advanced Transmission Electron Microscopy: Imaging and Diffraction Nanoscience Zuo, Jian Min; Spence, John C. H. https://doi.org/10.1007/978-1-4939-6607-3	book	January 2017
Scanning Electron Microscopy and X-Ray Microanalysis Goldstein, Joseph I.; Newbury, Dale E.; Michael, Joseph R. https://doi.org/10.1007/978-1-4939-6676-9	book	January 2018
High speed/low dose analytical electron microscopy with dynamic sampling Hujsak, Karl A.; Roth, Eric W.; Kellogg, William Micron, Vol. 108 https://doi.org/10.1016/j.micron.2018.03.001	journal	May 2018
Next-Generation Experimentation with Self-Driving Laboratories Häse, Florian; Roch, Loïc M.; Aspuru-Guzik, Alán Trends in Chemistry, Vol. 1, Issue 3 https://doi.org/10.1016/j.trechm.2019.02.007	journal	June 2019
Bayesian Optimization Garnett, Roman https://doi.org/10.1017/9781108348973	book	January 2023
High Dynamic Range Pixel Array Detector for Scanning Transmission Electron Microscopy Tate, Mark W.; Purohit, Prafull; Chamberlain, Darol Microscopy and Microanalysis, Vol. 22, Issue 1 https://doi.org/10.1017/S1431927615015664	journal	January 2016
High-Throughput Mass Spectrometry Imaging with Dynamic Sparse Sampling Hu, Hang; Helminiak, David; Yang, Manxi ACS Measurement Science Au, Vol. 2, Issue 5 https://doi.org/10.1021/acsmeasuresciau.2c00031	journal	August 2022
Autonomous Experiments in Scanning Probe Microscopy and Spectroscopy: Choosing Where to Explore Polarization Dynamics in Ferroelectrics Vasudevan, Rama K.; Kelley, Kyle P.; Hinkle, Jacob ACS Nano, Vol. 15, Issue 7 https://doi.org/10.1021/acsnano.0c10239	journal	July 2021
Automated and Autonomous Experiments in Electron and Scanning Probe Microscopy Kalinin, Sergei V.; Ziatdinov, Maxim; Hinkle, Jacob ACS Nano, Vol. 15, Issue 8 https://doi.org/10.1021/acsnano.1c02104	journal	July 2021
High-resolution non-destructive three-dimensional imaging of integrated circuits Holler, Mirko; Guizar-Sicairos, Manuel; Tsai, Esther H. R. Nature, Vol. 543, Issue 7645 https://doi.org/10.1038/nature21698	journal	March 2017
Off-the-shelf deep learning is not enough, and requires parsimony, Bayesianity, and causality Vasudevan, Rama K.; Ziatdinov, Maxim; Vlcek, Lukas npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-020-00487-0	journal	January 2021
AutoPhaseNN: unsupervised physics-aware deep learning of 3D nanoscale Bragg coherent diffraction imaging Yao, Yudong; Chan, Henry; Sankaranarayanan, Subramanian npj Computational Materials, Vol. 8, Issue 1 https://doi.org/10.1038/s41524-022-00803-w	journal	June 2022
A mobile robotic chemist Burger, Benjamin; Maffettone, Phillip M.; Gusev, Vladimir V. Nature, Vol. 583, Issue 7815 https://doi.org/10.1038/s41586-020-2442-2	journal	July 2020
A Kriging-Based Approach to Autonomous Experimentation with Applications to X-Ray Scattering Noack, Marcus M.; Yager, Kevin G.; Fukuto, Masafumi Scientific Reports, Vol. 9, Issue 1 https://doi.org/10.1038/s41598-019-48114-3	journal	August 2019
Advances in Kriging-Based Autonomous X-Ray Scattering Experiments Noack, Marcus M.; Doerk, Gregory S.; Li, Ruipeng Scientific Reports, Vol. 10, Issue 1 https://doi.org/10.1038/s41598-020-57887-x	journal	January 2020
Deep reinforcement learning for data-driven adaptive scanning in ptychography Schloz, Marcel; Müller, Johannes; Pekin, Thomas C. Scientific Reports, Vol. 13, Issue 1 https://doi.org/10.1038/s41598-023-35740-1	journal	May 2023
Gaussian processes for autonomous data acquisition at large-scale synchrotron and neutron facilities Noack, Marcus M.; Zwart, Petrus H.; Ushizima, Daniela M. Nature Reviews Physics, Vol. 3, Issue 10 https://doi.org/10.1038/s42254-021-00345-y	journal	July 2021
AI-enabled high-resolution scanning coherent diffraction imaging Cherukara, Mathew J.; Zhou, Tao; Nashed, Youssef Applied Physics Letters, Vol. 117, Issue 4 https://doi.org/10.1063/5.0013065	journal	July 2020
Rapid 3D nanoscale coherent imaging via physics-aware deep learning Chan, Henry; Nashed, Youssef S. G.; Kandel, Saugat Applied Physics Reviews, Vol. 8, Issue 2 https://doi.org/10.1063/5.0031486	journal	June 2021
Achieving high spatial resolution in a large field-of-view using lensless x-ray imaging Jiang, Yi; Deng, Junjing; Yao, Yudong Applied Physics Letters, Vol. 119, Issue 12 https://doi.org/10.1063/5.0067197	journal	September 2021
Sampling with Hammersley and Halton Points Wong, Tien-Tsin; Luk, Wai-Shing; Heng, Pheng-Ann Journal of Graphics Tools, Vol. 2, Issue 2 https://doi.org/10.1080/10867651.1997.10487471	journal	January 1997
Maia X-ray fluorescence imaging: Capturing detail in complex natural samples Ryan, C. G.; Siddons, D. P.; Kirkham, R. Journal of Physics: Conference Series, Vol. 499 https://doi.org/10.1088/1742-6596/499/1/012002	journal	April 2014
Probe microscopy is all you need ^* Kalinin, Sergei V.; Vasudevan, Rama; Liu, Yongtao Machine Learning: Science and Technology, Vol. 4, Issue 2 https://doi.org/10.1088/2632-2153/acccd5	journal	May 2023
Development of an on-the-fly data processing with information lossless compression for CITIUS detectors at SPring-8 Hiraki, Toshiyuki Nishiyama; Abe, Toshinori; Yamaga, Mitsuhiro Acta Crystallographica Section A Foundations and Advances, Vol. 77, Issue a2 https://doi.org/10.1107/S0108767321091583	journal	August 2021
A hard X-ray nanoprobe beamline for nanoscale microscopy Winarski, Robert P.; Holt, Martin V.; Rose, Volker Journal of Synchrotron Radiation, Vol. 19, Issue 6 https://doi.org/10.1107/S0909049512036783	journal	September 2012
Upscaling X-ray nanoimaging to macroscopic specimens Du, Ming; Di, Zichao; Gürsoy, Dogˇa Journal of Applied Crystallography, Vol. 54, Issue 2 https://doi.org/10.1107/S1600576721000194	journal	February 2021
Dynamic X-ray diffraction sampling for protein crystal positioning Scarborough, Nicole M.; Godaliyadda, G. M. Dilshan P.; Ye, Dong Hye Journal of Synchrotron Radiation, Vol. 24, Issue 1 https://doi.org/10.1107/S160057751601612X	journal	January 2017
Reinforcement Learning for Adaptive Illumination with X-rays Betterton, Jean-Raymond; Ratner, Daniel; Webb, Samuel 2020 IEEE International Conference on Robotics and Automation (ICRA) https://doi.org/10.1109/ICRA40945.2020.9196614	conference	May 2020
Dynamic Image Sampling Using a Novel Variance Based Probability Mass Function Grosche, Simon; Koller, Michael; Seiler, Jurgen IEEE Transactions on Computational Imaging, Vol. 6 https://doi.org/10.1109/TCI.2020.3031077	journal	January 2020
When Gaussian Process Meets Big Data: A Review of Scalable GPs Liu, Haitao; Ong, Yew-Soon; Shen, Xiaobo IEEE Transactions on Neural Networks and Learning Systems, Vol. 31, Issue 11 https://doi.org/10.1109/TNNLS.2019.2957109	journal	November 2020
On Surface Completion and Image Inpainting by Biharmonic Functions: Numerical Aspects Damelin, S. B.; Hoang, N. S. International Journal of Mathematics and Mathematical Sciences, Vol. 2018 https://doi.org/10.1155/2018/3950312	journal	January 2018
A Supervised Learning Approach for Dynamic Sampling Godaliyadda, G. M. Dilshan; Ye, Dong Hye; Uchic, Michael D. Electronic Imaging, Vol. 2016, Issue 19 https://doi.org/10.2352/ISSN.2470-1173.2016.19.COIMG-153	journal	February 2016
Deep learning at the edge enables real-time streaming ptychographic imaging Babu, Anakha V.; Zhou, Tao; Kandel, Saugat arXiv https://doi.org/10.48550/arXiv.2209.09408	preprint	January 2022
Experiment dataset for the FAST paper Kandel, Saugat; Zhou, Tao; Babu, Anakha V. Zenodo https://doi.org/10.5281/zenodo.7939730	dataset	January 2023

Similar Records

FAST (FAST AUTONOMOUS SCANNING TOOLKIT)

Software · Wed Aug 16 20:00:00 EDT 2023 · OSTI ID:code-112308

Physics Discovery in Nanoplasmonic Systems via Autonomous Experiments in Scanning Transmission Electron Microscopy

Journal Article · Sun Nov 06 23:00:00 EST 2022 · Advanced Science · OSTI ID:1897253

Demonstration of an AI-driven workflow for autonomous high-resolution scanning microscopy

Citation Formats

References (34)

Similar Records

Related Subjects