skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels

Abstract

Crystalline materials exhibit long-range ordered lattice unit, within which resides nonperiodic structural features called defects. These crystallographic defects play a vital role in determining the physical and mechanical properties of a wide range of material systems. While computer vision has demonstrated success in recognizing feature patterns in images with well defined contrast, automated identification of nanometer scale crystallographic defects in electron micrographs governed by complex contrast mechanism is still a challenging task. Here, building upon an advanced defect imaging mode that offers high feature clarity, we introduce DefectSegNet - a new convolutional neural network (CNN) architecture that performs semantic segmentation of three common crystallographic defects in structural alloys: dislocation lines, precipitates and voids. Results from supervised training of a small set of high-quality defect images of steels show high pixel-wise accuracy across all three types of defects: 91.60 ± 1.77% on dislocations, 93.39 ± 1.00% on precipitates, and 98.85 ± 0.56% on voids. We discuss the sources of uncertainties in CNN prediction, and the training data in terms of feature density, representation and homogeneity and their effects on deep learning performance. Further defect quantification using DefectSegNet prediction outperforms human expert average, presenting a promising new workflow for fast and statisticallymore » meaningful quantification of materials defects.« less

Authors:
 [1];  [2];  [3]; ORCiD logo [1];  [4]; ORCiD logo [5]
  1. Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
  2. Western Washington Univ., Bellingham, WA (United States)
  3. Univ. of Connecticut, Storrs, CT (United States)
  4. Western Washington Univ., Bellingham, WA (United States); Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
  5. Pacific Northwest National Lab. (PNNL), Richland, WA (United States); Univ. of Connecticut, Storrs, CT (United States)
Publication Date:
Research Org.:
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Fusion Energy Sciences (FES) (SC-24)
OSTI Identifier:
1559992
Report Number(s):
PNNL-SA-146546
Journal ID: ISSN 2045-2322
Grant/Contract Number:  
AC05-76RL01830
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Scientific Reports
Additional Journal Information:
Journal Volume: 9; Journal Issue: 1; Journal ID: ISSN 2045-2322
Publisher:
Nature Publishing Group
Country of Publication:
United States
Language:
English
Subject:
36 MATERIALS SCIENCE; machine leaning; STEM Imaging; convolutional neural network; semantic segmentation

Citation Formats

Roberts, Graham, Haile, Simon, Sainju, Rajat, Edwards, Danny J., Hutchinson, Brian J., and Zhu, Yuanyuan. Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels. United States: N. p., 2019. Web. doi:10.1038/s41598-019-49105-0.
Roberts, Graham, Haile, Simon, Sainju, Rajat, Edwards, Danny J., Hutchinson, Brian J., & Zhu, Yuanyuan. Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels. United States. https://doi.org/10.1038/s41598-019-49105-0
Roberts, Graham, Haile, Simon, Sainju, Rajat, Edwards, Danny J., Hutchinson, Brian J., and Zhu, Yuanyuan. Wed . "Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels". United States. https://doi.org/10.1038/s41598-019-49105-0. https://www.osti.gov/servlets/purl/1559992.
@article{osti_1559992,
title = {Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels},
author = {Roberts, Graham and Haile, Simon and Sainju, Rajat and Edwards, Danny J. and Hutchinson, Brian J. and Zhu, Yuanyuan},
abstractNote = {Crystalline materials exhibit long-range ordered lattice unit, within which resides nonperiodic structural features called defects. These crystallographic defects play a vital role in determining the physical and mechanical properties of a wide range of material systems. While computer vision has demonstrated success in recognizing feature patterns in images with well defined contrast, automated identification of nanometer scale crystallographic defects in electron micrographs governed by complex contrast mechanism is still a challenging task. Here, building upon an advanced defect imaging mode that offers high feature clarity, we introduce DefectSegNet - a new convolutional neural network (CNN) architecture that performs semantic segmentation of three common crystallographic defects in structural alloys: dislocation lines, precipitates and voids. Results from supervised training of a small set of high-quality defect images of steels show high pixel-wise accuracy across all three types of defects: 91.60 ± 1.77% on dislocations, 93.39 ± 1.00% on precipitates, and 98.85 ± 0.56% on voids. We discuss the sources of uncertainties in CNN prediction, and the training data in terms of feature density, representation and homogeneity and their effects on deep learning performance. Further defect quantification using DefectSegNet prediction outperforms human expert average, presenting a promising new workflow for fast and statistically meaningful quantification of materials defects.},
doi = {10.1038/s41598-019-49105-0},
url = {https://www.osti.gov/biblio/1559992}, journal = {Scientific Reports},
issn = {2045-2322},
number = 1,
volume = 9,
place = {United States},
year = {2019},
month = {9}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Fully convolutional networks for semantic segmentation
conference, June 2015


Densely Connected Convolutional Networks
conference, July 2017


Classification of corrosion defects in NiAl bronze through image analysis
journal, November 2010


Enhancing radiation tolerance by controlling defect mobility and migration pathways in multicomponent single-phase alloys
journal, December 2016


Advanced Steel Microstructural Classification by Deep Learning Methods
journal, February 2018


The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
conference, July 2017


A Method for Detection and Classification of Glass Defects in Low Resolution Images
conference, August 2011


A review of semantic segmentation using deep neural networks
journal, November 2017


High Throughput Quantitative Metallography for Complex Microstructures Using Deep Learning: A Case Study in Ultrahigh Carbon Steel
journal, February 2019


Image driven machine learning methods for microstructure recognition
journal, October 2016


Steel defect classification with Max-Pooling Convolutional Neural Networks
conference, June 2012

  • Masci, Jonathan; Meier, Ueli; Ciresan, Dan
  • 2012 International Joint Conference on Neural Networks (IJCNN 2012 - Brisbane), The 2012 International Joint Conference on Neural Networks (IJCNN)
  • https://doi.org/10.1109/IJCNN.2012.6252468

A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images
journal, July 2018


Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation
journal, January 2016


The relationship between grain boundary structure, defect mobility and grain boundary sink efficiency
journal, March 2015


Diffraction contrast STEM of dislocations: Imaging and simulations
journal, August 2011


Dislocation movement through random arrays of obstacles
journal, October 1966


Control and Use of Defects in Materials
journal, August 1998


AxonDeepSeg: automatic axon and myelin segmentation from microscopy data using convolutional neural networks
journal, February 2018


A systematic error in the determination of dislocation densities in thin films
journal, September 1961


A Kinematical Theory of Diffraction Contrast of Electron Transmission Microscope Images of Dislocations and other Defects
journal, May 1960

  • Hirsch, P. B.; Howie, A.; Whelan, M. J.
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 252, Issue 1017
  • https://doi.org/10.1098/rsta.1960.0013

Phase stability and structural defects in high-temperature Mo–Si–B alloys
journal, October 2008


A comparative study of texture measures with classification based on featured distributions
journal, January 1996


Automated defect analysis in electron microscopic images
journal, July 2018


Towards bend-contour-free dislocation imaging via diffraction contrast STEM
journal, October 2018


Best practices for convolutional neural networks applied to visual document analysis
conference, January 2003


Image Enhancement via Subimage Histogram Equalization Based on Mean and Variance
journal, January 2017


Disconnections and other defects associated with twin interfaces
journal, October 2016


Survey on semantic image segmentation techniques
conference, December 2017


Importance-Aware Semantic Segmentation for Autonomous Vehicles
journal, January 2019


Analytical Transmission Electron Microscopy
journal, August 2005


A computer vision approach for automated analysis and classification of microstructural image data
journal, December 2015