Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images
Abstract
Cutting edge deep learning techniques allow for image segmentation with great speed and accuracy. However, application to problems in materials science is often difficult since these complex models may have difficultly learning meaningful image features that would enable extension to new datasets. In situ electron microscopy provides a clear platform for utilizing automated image analysis. In this work, we consider the case of studying coarsening dynamics in supported nanoparticles, which is important for understanding, for example, the degradation of industrial catalysts. By systematically studying dataset preparation, neural network architecture, and accuracy evaluation, we describe important considerations in applying deep learning to physical applications, where generalizable and convincing models are required. With a focus on unique challenges that arise in high-resolution images, we propose methods for optimizing performance of image segmentation using convolutional neural networks, critically examining the application of complex deep learning models in favor of motivating intentional process design.
- Authors:
-
[1];
[2];
[1]
- Univ. of Pennsylvania, Philadelphia, PA (United States)
- Brookhaven National Lab. (BNL), Upton, NY (United States)
- Univ. of Puerto Rico, San Juan, PR (United States)
- Publication Date:
- Research Org.:
- Brookhaven National Lab. (BNL), Upton, NY (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Basic Energy Sciences (BES); National Science Foundation (NSF)
- OSTI Identifier:
- 1677672
- Report Number(s):
- BNL-219976-2020-JAAM
Journal ID: ISSN 2057-3960
- Grant/Contract Number:
- SC0012704; 1809398.
- Resource Type:
- Journal Article: Accepted Manuscript
- Journal Name:
- npj Computational Materials
- Additional Journal Information:
- Journal Volume: 6; Journal Issue: 1; Journal ID: ISSN 2057-3960
- Publisher:
- Nature Publishing Group
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 36 MATERIALS SCIENCE
Citation Formats
Horwath, James P., Zakharov, Dmitri N., Mégret, Rémi, and Stach, Eric A. Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. United States: N. p., 2020.
Web. doi:10.1038/s41524-020-00363-x.
Horwath, James P., Zakharov, Dmitri N., Mégret, Rémi, & Stach, Eric A. Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. United States. https://doi.org/10.1038/s41524-020-00363-x
Horwath, James P., Zakharov, Dmitri N., Mégret, Rémi, and Stach, Eric A. Wed .
"Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images". United States. https://doi.org/10.1038/s41524-020-00363-x. https://www.osti.gov/servlets/purl/1677672.
@article{osti_1677672,
title = {Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images},
author = {Horwath, James P. and Zakharov, Dmitri N. and Mégret, Rémi and Stach, Eric A.},
abstractNote = {Cutting edge deep learning techniques allow for image segmentation with great speed and accuracy. However, application to problems in materials science is often difficult since these complex models may have difficultly learning meaningful image features that would enable extension to new datasets. In situ electron microscopy provides a clear platform for utilizing automated image analysis. In this work, we consider the case of studying coarsening dynamics in supported nanoparticles, which is important for understanding, for example, the degradation of industrial catalysts. By systematically studying dataset preparation, neural network architecture, and accuracy evaluation, we describe important considerations in applying deep learning to physical applications, where generalizable and convincing models are required. With a focus on unique challenges that arise in high-resolution images, we propose methods for optimizing performance of image segmentation using convolutional neural networks, critically examining the application of complex deep learning models in favor of motivating intentional process design.},
doi = {10.1038/s41524-020-00363-x},
url = {https://www.osti.gov/biblio/1677672},
journal = {npj Computational Materials},
issn = {2057-3960},
number = 1,
volume = 6,
place = {United States},
year = {2020},
month = {7}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journalFigures / Tables:
Fig. 1: UNet architecture improves particle segmentation compared to encoder–decoder architecture. Segmentation results for UNet-type architecture on 512 x 512 resolution images. a Raw output from the model overlaid on the raw image; notice the sharp activation cutoff at the particle edges. b Threshold applied to image to show finalmore »
All figures and tables
(8 total)
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Frontiers of in situ electron microscopy
journal, January 2015
- Zheng, Haimei; Meng, Ying Shirley; Zhu, Yimei
- MRS Bulletin, Vol. 40, Issue 1
In Situ Studies of Chemistry and Structure of Materials in Reactive Environments
journal, January 2011
- Tao, F.; Salmeron, M.
- Science, Vol. 331, Issue 6014
Current status and future directions for in situ transmission electron microscopy
journal, November 2016
- Taheri, Mitra L.; Stach, Eric A.; Arslan, Ilke
- Ultramicroscopy, Vol. 170
Materials science with large-scale data and informatics: Unlocking new opportunities
journal, May 2016
- Hill, Joanne; Mulholland, Gregory; Persson, Kristin
- MRS Bulletin, Vol. 41, Issue 5
Direct Observations of Oxygen-induced Platinum Nanoparticle Ripening Studied by In Situ TEM
journal, June 2010
- Simonsen, Søren B.; Chorkendorff, Ib; Dahl, Søren
- Journal of the American Chemical Society, Vol. 132, Issue 23
Deep learning for cellular image analysis
journal, May 2019
- Moen, Erick; Bannon, Dylan; Kudo, Takamasa
- Nature Methods, Vol. 16, Issue 12
Overfitting and undercomputing in machine learning
journal, September 1995
- Dietterich, Tom
- ACM Computing Surveys, Vol. 27, Issue 3
A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images
journal, July 2018
- Madsen, Jacob; Liu, Pei; Kling, Jens
- Advanced Theory and Simulations, Vol. 1, Issue 8
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations
journal, October 2016
- Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem
- ACS Nano, Vol. 11, Issue 12
Towards Real Time Quantitative Analysis of Supported Nanoparticle Ensemble Evolution Investigated by Environmental TEM
journal, August 2018
- Zakharov, Dmitri N.; Lin, Yuewei; Megret, Remi
- Microscopy and Microanalysis, Vol. 24, Issue S1
Sintering of Catalytic Nanoparticles: Particle Migration or Ostwald Ripening?
journal, April 2013
- Hansen, Thomas W.; DeLaRiva, Andrew T.; Challa, Sivakumar R.
- Accounts of Chemical Research, Vol. 46, Issue 8
Deep Learning in Medical Image Analysis
journal, June 2017
- Shen, Dinggang; Wu, Guorong; Suk, Heung-Il
- Annual Review of Biomedical Engineering, Vol. 19, Issue 1
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
journal, December 2017
- Badrinarayanan, Vijay; Kendall, Alex; Cipolla, Roberto
- IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 12
Fully convolutional networks for semantic segmentation
conference, June 2015
- Long, Jonathan; Shelhamer, Evan; Darrell, Trevor
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
U-Net: Convolutional Networks for Biomedical Image Segmentation
book, November 2015
- Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas
- Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III
Kernel-based methods and function approximation
conference, January 2001
- Baudat, G.; Anouar, F.
- International Joint Conference on Neural Networks (IJCNN'01), IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222)
A Threshold Selection Method from Gray-Level Histograms
journal, January 1979
- Otsu, Nobuyuki
- IEEE Transactions on Systems, Man, and Cybernetics, Vol. 9, Issue 1
Characteristic analysis of Otsu threshold and its applications
journal, May 2011
- Xu, Xiangyang; Xu, Shengzhou; Jin, Lianghai
- Pattern Recognition Letters, Vol. 32, Issue 7
On the size of Convolutional Neural Networks and generalization performance
conference, December 2016
- Kabkab, Maya; Hand, Emily; Chellappa, Rama
- 2016 23rd International Conference on Pattern Recognition (ICPR)
scikit-image: image processing in Python
journal, January 2014
- van der Walt, Stéfan; Schönberger, Johannes L.; Nunez-Iglesias, Juan
- PeerJ, Vol. 2
Figures / Tables found in this record:
Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.