skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images

Abstract

Cutting edge deep learning techniques allow for image segmentation with great speed and accuracy. However, application to problems in materials science is often difficult since these complex models may have difficultly learning meaningful image features that would enable extension to new datasets. In situ electron microscopy provides a clear platform for utilizing automated image analysis. In this work, we consider the case of studying coarsening dynamics in supported nanoparticles, which is important for understanding, for example, the degradation of industrial catalysts. By systematically studying dataset preparation, neural network architecture, and accuracy evaluation, we describe important considerations in applying deep learning to physical applications, where generalizable and convincing models are required. With a focus on unique challenges that arise in high-resolution images, we propose methods for optimizing performance of image segmentation using convolutional neural networks, critically examining the application of complex deep learning models in favor of motivating intentional process design.

Authors:
ORCiD logo [1]; ORCiD logo [2];  [3]; ORCiD logo [1]
  1. Univ. of Pennsylvania, Philadelphia, PA (United States)
  2. Brookhaven National Lab. (BNL), Upton, NY (United States)
  3. Univ. of Puerto Rico, San Juan, PR (United States)
Publication Date:
Research Org.:
Brookhaven National Lab. (BNL), Upton, NY (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Basic Energy Sciences (BES); National Science Foundation (NSF)
OSTI Identifier:
1677672
Report Number(s):
BNL-219976-2020-JAAM
Journal ID: ISSN 2057-3960
Grant/Contract Number:  
SC0012704; 1809398.
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
npj Computational Materials
Additional Journal Information:
Journal Volume: 6; Journal Issue: 1; Journal ID: ISSN 2057-3960
Publisher:
Nature Publishing Group
Country of Publication:
United States
Language:
English
Subject:
36 MATERIALS SCIENCE

Citation Formats

Horwath, James P., Zakharov, Dmitri N., Mégret, Rémi, and Stach, Eric A. Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. United States: N. p., 2020. Web. doi:10.1038/s41524-020-00363-x.
Horwath, James P., Zakharov, Dmitri N., Mégret, Rémi, & Stach, Eric A. Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. United States. https://doi.org/10.1038/s41524-020-00363-x
Horwath, James P., Zakharov, Dmitri N., Mégret, Rémi, and Stach, Eric A. Wed . "Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images". United States. https://doi.org/10.1038/s41524-020-00363-x. https://www.osti.gov/servlets/purl/1677672.
@article{osti_1677672,
title = {Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images},
author = {Horwath, James P. and Zakharov, Dmitri N. and Mégret, Rémi and Stach, Eric A.},
abstractNote = {Cutting edge deep learning techniques allow for image segmentation with great speed and accuracy. However, application to problems in materials science is often difficult since these complex models may have difficultly learning meaningful image features that would enable extension to new datasets. In situ electron microscopy provides a clear platform for utilizing automated image analysis. In this work, we consider the case of studying coarsening dynamics in supported nanoparticles, which is important for understanding, for example, the degradation of industrial catalysts. By systematically studying dataset preparation, neural network architecture, and accuracy evaluation, we describe important considerations in applying deep learning to physical applications, where generalizable and convincing models are required. With a focus on unique challenges that arise in high-resolution images, we propose methods for optimizing performance of image segmentation using convolutional neural networks, critically examining the application of complex deep learning models in favor of motivating intentional process design.},
doi = {10.1038/s41524-020-00363-x},
url = {https://www.osti.gov/biblio/1677672}, journal = {npj Computational Materials},
issn = {2057-3960},
number = 1,
volume = 6,
place = {United States},
year = {2020},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Figures / Tables:

Fig. 1 Fig. 1: UNet architecture improves particle segmentation compared to encoder–decoder architecture. Segmentation results for UNet-type architecture on 512 x 512 resolution images. a Raw output from the model overlaid on the raw image; notice the sharp activation cutoff at the particle edges. b Threshold applied to image to show finalmore » segmentation result. Yellow arrows indicate small particles that were successfully recognized. Scale bar represents 50 nm.« less

Save / Share:

Works referenced in this record:

Frontiers of in situ electron microscopy
journal, January 2015


Current status and future directions for in situ transmission electron microscopy
journal, November 2016


Materials science with large-scale data and informatics: Unlocking new opportunities
journal, May 2016


Direct Observations of Oxygen-induced Platinum Nanoparticle Ripening Studied by In Situ TEM
journal, June 2010


Deep learning for cellular image analysis
journal, May 2019


Overfitting and undercomputing in machine learning
journal, September 1995


A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images
journal, July 2018


Towards Real Time Quantitative Analysis of Supported Nanoparticle Ensemble Evolution Investigated by Environmental TEM
journal, August 2018


Sintering of Catalytic Nanoparticles: Particle Migration or Ostwald Ripening?
journal, April 2013


Deep Learning in Medical Image Analysis
journal, June 2017


SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
journal, December 2017


Fully convolutional networks for semantic segmentation
conference, June 2015


U-Net: Convolutional Networks for Biomedical Image Segmentation
book, November 2015

  • Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas
  • Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III
  • https://doi.org/10.1007/978-3-319-24574-4_28

Kernel-based methods and function approximation
conference, January 2001

  • Baudat, G.; Anouar, F.
  • International Joint Conference on Neural Networks (IJCNN'01), IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222)
  • https://doi.org/10.1109/IJCNN.2001.939539

A Threshold Selection Method from Gray-Level Histograms
journal, January 1979


Characteristic analysis of Otsu threshold and its applications
journal, May 2011


On the size of Convolutional Neural Networks and generalization performance
conference, December 2016


scikit-image: image processing in Python
journal, January 2014


    Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.