Rapid and flexible segmentation of electron microscopy data using few-shot machine learning

Akers, Sarah; Kautz, Elizabeth; Trevino-Gavito, Andrea; Olszta, Matthew; Matthews, Bethany E.; Wang, Le; Du, Yingge; Spurgeon, Steven R.

doi:10.1038/s41524-021-00652-z

Title: Rapid and flexible segmentation of electron microscopy data using few-shot machine learning

Journal Article · 16 November 2021 · npj Computational Materials

DOI: https://doi.org/10.1038/s41524-021-00652-z · OSTI ID:1830837

; Kautz, Elizabeth; Trevino-Gavito, Andrea; Olszta, Matthew; Matthews, Bethany E.; Wang, Le; Du, Yingge;

Abstract Automatic segmentation of key microstructural features in atomic-scale electron microscope images is critical to improved understanding of structure–property relationships in many important materials and chemical systems. However, the present paradigm involves time-intensive manual analysis that is inherently biased, error-prone, and unable to accommodate the large volumes of data produced by modern instrumentation. While more automated approaches have been proposed, many are not robust to a high variety of data, and do not generalize well to diverse microstructural features and material systems. Here, we present a flexible, semi-supervised few-shot machine learning approach for segmentation of scanning transmission electron microscopy images of three oxide material systems: (1) epitaxial heterostructures of SrTiO ₃ /Ge, (2) La _0.8 Sr _0.2 FeO ₃ thin films, and (3) MoO ₃ nanoparticles. We demonstrate that the few-shot learning method is more robust against noise, more reconfigurable, and requires less data than conventional image analysis methods. This approach can enable rapid image classification and microstructural feature mapping needed for emerging high-throughput characterization and autonomous microscope platforms.

View Journal Article

Cite

Export

Save

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE; USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE Office of Science (SC), Basic Energy Sciences (BES). Materials Sciences & Engineering Division

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 1830837

Report Number(s):: PNNL-SA--160786; 187; PII: 652

Journal Information:: npj Computational Materials, Journal Name: npj Computational Materials Journal Issue: 1 Vol. 7; ISSN 2057-3960

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (53)

Interface Engineering and Emergent Phenomena in Oxide Heterostructures Huang, Zhen; Renshaw Wang, Xiao Advanced Materials, Vol. 30, Issue 47 https://doi.org/10.1002/adma.201802439	journal	August 2018
A Convolutional Neural Network Method for Boundary Optimization Enables Few-Shot Learning for Biomedical Image Segmentation Rutter, Erica M.; Lagergren, John H.; Flores, Kevin B. Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data https://doi.org/10.1007/978-3-030-33391-1_22	book	January 2019
U-Net: Convolutional Networks for Biomedical Image Segmentation Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III https://doi.org/10.1007/978-3-319-24574-4_28	book	November 2015
Contrast Limited Adaptive Histogram Equalization Zuiderveld, Karel Graphics Gems https://doi.org/10.1016/B978-0-12-336156-1.50061-6	book	January 1994
Adaptive histogram equalization and its variations Pizer, Stephen M.; Amburn, E. Philip; Austin, John D. Computer Vision, Graphics, and Image Processing, Vol. 39, Issue 3 https://doi.org/10.1016/S0734-189X(87)80186-X	journal	September 1987
A computer vision approach for automated analysis and classification of microstructural image data DeCost, Brian L.; Holm, Elizabeth A. Computational Materials Science, Vol. 110 https://doi.org/10.1016/j.commatsci.2015.08.011	journal	December 2015
Image driven machine learning methods for microstructure recognition Chowdhury, Aritra; Kautz, Elizabeth; Yener, Bülent Computational Materials Science, Vol. 123 https://doi.org/10.1016/j.commatsci.2016.05.034	journal	October 2016
A machine learning approach to thermal conductivity modeling: A case study on irradiated uranium-molybdenum nuclear fuels Kautz, Elizabeth J.; Hagen, Alexander R.; Johns, Jesse M. Computational Materials Science, Vol. 161 https://doi.org/10.1016/j.commatsci.2019.01.044	journal	April 2019
Adaptive characterization of microstructure dataset using a two stage machine learning approach Baskaran, Arun; Kane, Genevieve; Biggs, Krista Computational Materials Science, Vol. 177 https://doi.org/10.1016/j.commatsci.2020.109593	journal	May 2020
Informatics and data science in materials microscopy Voyles, Paul M. Current Opinion in Solid State and Materials Science, Vol. 21, Issue 3 https://doi.org/10.1016/j.cossms.2016.10.001	journal	June 2017
An image-driven machine learning approach to kinetic modeling of a discontinuous precipitation reaction Kautz, Elizabeth; Ma, Wufei; Jana, Saumyadeep Materials Characterization, Vol. 166 https://doi.org/10.1016/j.matchar.2020.110379	journal	August 2020
Race against the Machine: can deep learning recognize microstructures as well as the trained human eye? Larmuseau, Michiel; Sluydts, Michael; Theuwissen, Koenraad Scripta Materialia, Vol. 193 https://doi.org/10.1016/j.scriptamat.2020.10.026	journal	March 2021
The impact of STEM aberration correction on materials science Pennycook, Stephen J. Ultramicroscopy, Vol. 180 https://doi.org/10.1016/j.ultramic.2017.03.020	journal	September 2017
High Throughput Quantitative Metallography for Complex Microstructures Using Deep Learning: A Case Study in Ultrahigh Carbon Steel DeCost, Brian L.; Lei, Bo; Francis, Toby Microscopy and Microanalysis, Vol. 25, Issue 1 https://doi.org/10.1017/S1431927618015635	journal	February 2019
“You Are Not My Type”: An Evaluation of Classification Methods for Automatic Phytolith Identification Díez-Pastor, José-Francisco; Latorre-Carmona, Pedro; Arnaiz-González, Álvar Microscopy and Microanalysis, Vol. 26, Issue 6 https://doi.org/10.1017/S1431927620024629	journal	November 2020
Phases and Interfaces from Real Space Atomically Resolved Data: Physics-Based Deep Data Image Analysis Vasudevan, Rama K.; Ziatdinov, Maxim; Jesse, Stephen Nano Letters, Vol. 16, Issue 9 https://doi.org/10.1021/acs.nanolett.6b02130	journal	August 2016
Low Data Drug Discovery with One-Shot Learning Altae-Tran, Han; Ramsundar, Bharath; Pappu, Aneesh S. ACS Central Science, Vol. 3, Issue 4 https://doi.org/10.1021/acscentsci.6b00367	journal	April 2017
Knowledge Extraction from Atomically Resolved Images Vlcek, Lukas; Maksov, Artem; Pan, Minghu ACS Nano, Vol. 11, Issue 10 https://doi.org/10.1021/acsnano.7b05036	journal	October 2017
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem ACS Nano, Vol. 11, Issue 12 https://doi.org/10.1021/acsnano.7b07504	journal	October 2016
Pictorial Structures for Object Recognition Felzenszwalb, Pedro F.; Huttenlocher, Daniel P. International Journal of Computer Vision, Vol. 61, Issue 1 https://doi.org/10.1023/B:VISI.0000042934.15159.49	journal	January 2005
1,500 scientists lift the lid on reproducibility Baker, Monya Nature, Vol. 533, Issue 7604 https://doi.org/10.1038/533452a	journal	May 2016
Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2 Maksov, Artem; Dyck, Ondrej; Wang, Kai npj Computational Materials, Vol. 5, Issue 1 https://doi.org/10.1038/s41524-019-0152-9	journal	February 2019
Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images Horwath, James P.; Zakharov, Dmitri N.; Mégret, Rémi npj Computational Materials, Vol. 6, Issue 1 https://doi.org/10.1038/s41524-020-00363-x	journal	July 2020
Probing atomic-scale symmetry breaking by rotationally invariant machine learning of multidimensional electron scattering Oxley, Mark P.; Ziatdinov, Maxim; Dyck, Ondrej npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00527-3	journal	May 2021
Towards data-driven next-generation transmission electron microscopy Spurgeon, Steven R.; Ophus, Colin; Jones, Lewys Nature Materials https://doi.org/10.1038/s41563-020-00833-z	journal	October 2020
The Hessian Blob Algorithm: Precise Particle Detection in Atomic Force Microscopy Imagery Marsh, Brendan P.; Chada, Nagaraju; Sanganna Gari, Raghavendar Reddy Scientific Reports, Vol. 8, Issue 1 https://doi.org/10.1038/s41598-018-19379-x	journal	January 2018
Advanced Steel Microstructural Classification by Deep Learning Methods Azimi, Seyed Majid; Britz, Dominik; Engstler, Michael Scientific Reports, Vol. 8, Issue 1 https://doi.org/10.1038/s41598-018-20037-5	journal	February 2018
Plasma Hsp90 levels in patients with systemic sclerosis and relation to lung and skin involvement: a cross-sectional and longitudinal study Štorkánová, Hana; Oreská, Sabína; Špiritović, Maja Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-020-79139-8	journal	January 2021
Oxygen vacancies: The (in)visible friend of oxide electronics Gunkel, F.; Christensen, D. V.; Chen, Y. Z. Applied Physics Letters, Vol. 116, Issue 12 https://doi.org/10.1063/1.5143309	journal	March 2020
Image-driven discriminative and generative machine learning algorithms for establishing microstructure–processing relationships Ma, W.; Kautz, E. J.; Baskaran, A. Journal of Applied Physics, Vol. 128, Issue 13 https://doi.org/10.1063/5.0013720	journal	October 2020
A mixed-scale dense convolutional neural network for image analysis Pelt, Daniël M.; Sethian, James A. Proceedings of the National Academy of Sciences, Vol. 115, Issue 2 https://doi.org/10.1073/pnas.1715832114	journal	December 2017
Ultra-high resolution electron microscopy Oxley, Mark P.; Lupini, Andrew R.; Pennycook, Stephen J. Reports on Progress in Physics, Vol. 80, Issue 2 https://doi.org/10.1088/1361-6633/80/2/026101	journal	December 2016
Deep learning in electron microscopy Ede, Jeffrey M. Machine Learning: Science and Technology, Vol. 2, Issue 1 https://doi.org/10.1088/2632-2153/abd614	journal	March 2021
Layer-resolved band bending at the n − SrTi O 3 ( 001 ) / p − Ge ( 001 ) interface Du, Y.; Sushko, P. V.; Spurgeon, S. R. Physical Review Materials, Vol. 2, Issue 9 https://doi.org/10.1103/PhysRevMaterials.2.094602	journal	September 2018
Hole-induced electronic and optical transitions in L a 1 − x S r x Fe O 3 epitaxial thin films Wang, Le; Du, Yingge; Sushko, Peter V. Physical Review Materials, Vol. 3, Issue 2 https://doi.org/10.1103/PhysRevMaterials.3.025401	journal	February 2019
ImageNet: A large-scale hierarchical image database Deng, Jia; Dong, Wei; Socher, Richard 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), 2009 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2009.5206848	conference	June 2009
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Learning to Compare: Relation Network for Few-Shot Learning Sung, Flood; Yang, Yongxin; Zhang, Li 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2018.00131	conference	June 2018
A Unified Approach for Conventional Zero-Shot, Generalized Zero-Shot, and Few-Shot Learning Rahman, Shafin; Khan, Salman; Porikli, Fatih IEEE Transactions on Image Processing, Vol. 27, Issue 11 https://doi.org/10.1109/TIP.2018.2861573	journal	November 2018
SLIC Superpixels Compared to State-of-the-Art Superpixel Methods Achanta, R.; Shaji, A.; Smith, K. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, Issue 11, p. 2274-2282 https://doi.org/10.1109/TPAMI.2012.120	journal	November 2012
A Threshold Selection Method from Gray-Level Histograms Otsu, Nobuyuki IEEE Transactions on Systems, Man, and Cybernetics, Vol. 9, Issue 1 https://doi.org/10.1109/TSMC.1979.4310076	journal	January 1979
Decoding crystallography from high-resolution electron imaging and diffraction datasets with deep learning Aguiar, J. A.; Gong, M. L.; Unocic, R. R. Science Advances, Vol. 5, Issue 10 https://doi.org/10.1126/sciadv.aaw1949	journal	October 2019
Exploring order parameters and dynamic processes in disordered systems via variational autoencoders Kalinin, Sergei V.; Dyck, Ondrej; Jesse, Stephen Science Advances, Vol. 7, Issue 17 https://doi.org/10.1126/sciadv.abd5084	journal	April 2021
Point Defects in Oxides: Tailoring Materials Through Defect Engineering Tuller, Harry L.; Bishop, Sean R. Annual Review of Materials Research, Vol. 41, Issue 1 https://doi.org/10.1146/annurev-matsci-062910-100442	journal	August 2011
Materials Data Science: Current Status and Future Outlook Kalidindi, Surya R.; De Graef, Marc Annual Review of Materials Research, Vol. 45, Issue 1 https://doi.org/10.1146/annurev-matsci-070214-020844	journal	July 2015
Materials Characterization in the Aberration-Corrected Scanning Transmission Electron Microscope Varela, M.; Lupini, A. R.; Benthem, K. van Annual Review of Materials Research, Vol. 35, Issue 1 https://doi.org/10.1146/annurev.matsci.35.102103.090513	journal	August 2005
Aberration-corrected scanning transmission electron microscopy for atomic-resolution studies of functional oxides MacLaren, Ian; Ramasse, Quentin M. International Materials Reviews, Vol. 59, Issue 3 https://doi.org/10.1179/1743280413Y.0000000026	journal	March 2014
Smart Align—a new tool for robust non-rigid registration of scanning microscope data Jones, Lewys; Yang, Hao; Pennycook, Timothy J. Advanced Structural and Chemical Imaging, Vol. 1, Issue 1 https://doi.org/10.1186/s40679-015-0008-4	journal	July 2015
Detection of defects in atomic-resolution images of materials using cycle analysis Ovchinnikov, Oleg S.; O’Hara, Andrew; Jesse, Stephen Advanced Structural and Chemical Imaging, Vol. 6, Issue 1 https://doi.org/10.1186/s40679-020-00070-x	journal	March 2020
Materials Advances through Aberration-Corrected Electron Microscopy Pennycook, S. J.; Varela, M.; Hetherington, C. J. D. MRS Bulletin, Vol. 31, Issue 1 https://doi.org/10.1557/mrs2006.4	journal	January 2006
Microstructure Instance Segmentation from Aluminum Alloy Metallographic Image Using Different Loss Functions Chen, Dali; Guo, Dinghao; Liu, Shixin Symmetry, Vol. 12, Issue 4 https://doi.org/10.3390/sym12040639	journal	April 2020
Supplemental Data for Rapid and Flexible Segmentation of Electron Microscopy Data Using Few-Shot Machine Learning Akers, Sarah; Kautz, Elizabeth; Trevino-Gavito, Andrea figshare https://doi.org/10.6084/m9.figshare.14850102.v2	dataset	January 2021
scikit-image: image processing in Python van der Walt, Stéfan; Schönberger, Johannes L.; Nunez-Iglesias, Juan PeerJ, Vol. 2 https://doi.org/10.7717/peerj.453	journal	January 2014

Similar Records

Multimodal Few-Shot Segmentation of Electron Micrographs

Technical Report · 2023 · OSTI ID:3363323

Meyers, Eli P.; Bilbrey, Jenna A.; Matthews, Bethany E.; +2 more

Few-shot leaf segmentation

Software · 2023 · OSTI ID:code-103229

Lagergren, John

Design of a graphical user interface for few-shot machine learning classification of electron microscopy data

Journal Article · 2022 · Computational Materials Science · OSTI ID:1838120

Doty, Christina; Gallagher, Shaun; Cui, Wenqi; +5 more

Related Subjects

36 MATERIALS SCIENCE
few shot learning
machine leaning
scanning transmission electron microscopy (STEM)
segmentation

Title: Rapid and flexible segmentation of electron microscopy data using few-shot machine learning

Citation Formats

References (53)

Similar Records

Related Subjects