Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels
Abstract
Crystalline materials exhibit long-range ordered lattice unit, within which resides nonperiodic structural features called defects. These crystallographic defects play a vital role in determining the physical and mechanical properties of a wide range of material systems. While computer vision has demonstrated success in recognizing feature patterns in images with well defined contrast, automated identification of nanometer scale crystallographic defects in electron micrographs governed by complex contrast mechanism is still a challenging task. Here, building upon an advanced defect imaging mode that offers high feature clarity, we introduce DefectSegNet - a new convolutional neural network (CNN) architecture that performs semantic segmentation of three common crystallographic defects in structural alloys: dislocation lines, precipitates and voids. Results from supervised training of a small set of high-quality defect images of steels show high pixel-wise accuracy across all three types of defects: 91.60 ± 1.77% on dislocations, 93.39 ± 1.00% on precipitates, and 98.85 ± 0.56% on voids. We discuss the sources of uncertainties in CNN prediction, and the training data in terms of feature density, representation and homogeneity and their effects on deep learning performance. Further defect quantification using DefectSegNet prediction outperforms human expert average, presenting a promising new workflow for fast and statisticallymore »
- Authors:
-
[1];
[5]
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Western Washington Univ., Bellingham, WA (United States)
- Univ. of Connecticut, Storrs, CT (United States)
- Western Washington Univ., Bellingham, WA (United States); Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States); Univ. of Connecticut, Storrs, CT (United States)
- Publication Date:
- Research Org.:
- Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Fusion Energy Sciences (FES)
- OSTI Identifier:
- 1559992
- Report Number(s):
- PNNL-SA-146546
Journal ID: ISSN 2045-2322
- Grant/Contract Number:
- AC05-76RL01830
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Scientific Reports
- Additional Journal Information:
- Journal Volume: 9; Journal Issue: 1; Journal ID: ISSN 2045-2322
- Publisher:
- Nature Publishing Group
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 36 MATERIALS SCIENCE; machine leaning; STEM Imaging; convolutional neural network; semantic segmentation
Citation Formats
Roberts, Graham, Haile, Simon, Sainju, Rajat, Edwards, Danny J., Hutchinson, Brian J., and Zhu, Yuanyuan. Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels. United States: N. p., 2019.
Web. doi:10.1038/s41598-019-49105-0.
Roberts, Graham, Haile, Simon, Sainju, Rajat, Edwards, Danny J., Hutchinson, Brian J., & Zhu, Yuanyuan. Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels. United States. https://doi.org/10.1038/s41598-019-49105-0
Roberts, Graham, Haile, Simon, Sainju, Rajat, Edwards, Danny J., Hutchinson, Brian J., and Zhu, Yuanyuan. Wed .
"Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels". United States. https://doi.org/10.1038/s41598-019-49105-0. https://www.osti.gov/servlets/purl/1559992.
@article{osti_1559992,
title = {Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels},
author = {Roberts, Graham and Haile, Simon and Sainju, Rajat and Edwards, Danny J. and Hutchinson, Brian J. and Zhu, Yuanyuan},
abstractNote = {Crystalline materials exhibit long-range ordered lattice unit, within which resides nonperiodic structural features called defects. These crystallographic defects play a vital role in determining the physical and mechanical properties of a wide range of material systems. While computer vision has demonstrated success in recognizing feature patterns in images with well defined contrast, automated identification of nanometer scale crystallographic defects in electron micrographs governed by complex contrast mechanism is still a challenging task. Here, building upon an advanced defect imaging mode that offers high feature clarity, we introduce DefectSegNet - a new convolutional neural network (CNN) architecture that performs semantic segmentation of three common crystallographic defects in structural alloys: dislocation lines, precipitates and voids. Results from supervised training of a small set of high-quality defect images of steels show high pixel-wise accuracy across all three types of defects: 91.60 ± 1.77% on dislocations, 93.39 ± 1.00% on precipitates, and 98.85 ± 0.56% on voids. We discuss the sources of uncertainties in CNN prediction, and the training data in terms of feature density, representation and homogeneity and their effects on deep learning performance. Further defect quantification using DefectSegNet prediction outperforms human expert average, presenting a promising new workflow for fast and statistically meaningful quantification of materials defects.},
doi = {10.1038/s41598-019-49105-0},
journal = {Scientific Reports},
number = 1,
volume = 9,
place = {United States},
year = {Wed Sep 04 00:00:00 EDT 2019},
month = {Wed Sep 04 00:00:00 EDT 2019}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 57 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Enhancing radiation tolerance by controlling defect mobility and migration pathways in multicomponent single-phase alloys
journal, December 2016
- Lu, Chenyang; Niu, Liangliang; Chen, Nanjun
- Nature Communications, Vol. 7, Issue 1
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
conference, July 2017
- Jegou, Simon; Drozdzal, Michal; Vazquez, David
- 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
High Throughput Quantitative Metallography for Complex Microstructures Using Deep Learning: A Case Study in Ultrahigh Carbon Steel
journal, February 2019
- DeCost, Brian L.; Lei, Bo; Francis, Toby
- Microscopy and Microanalysis, Vol. 25, Issue 1
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations
journal, October 2016
- Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem
- ACS Nano, Vol. 11, Issue 12
Fully Convolutional Networks for Semantic Segmentation
preprint, January 2016
- Shelhamer, Evan; Long, Jonathan; Darrell, Trevor
- arXiv
Advanced Steel Microstructural Classification by Deep Learning Methods
text, January 2017
- Azimi, Seyed Majid; Britz, Dominik; Engstler, Michael
- arXiv
Image driven machine learning methods for microstructure recognition
journal, October 2016
- Chowdhury, Aritra; Kautz, Elizabeth; Yener, Bülent
- Computational Materials Science, Vol. 123
Steel defect classification with Max-Pooling Convolutional Neural Networks
conference, June 2012
- Masci, Jonathan; Meier, Ueli; Ciresan, Dan
- 2012 International Joint Conference on Neural Networks (IJCNN 2012 - Brisbane), The 2012 International Joint Conference on Neural Networks (IJCNN)
Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation
journal, January 2016
- Zhu, Hongyuan; Meng, Fanman; Cai, Jianfei
- Journal of Visual Communication and Image Representation, Vol. 34
Dislocation movement through random arrays of obstacles
journal, October 1966
- Foreman, A. J. E.; Makin, M. J.
- Philosophical Magazine, Vol. 14, Issue 131
Phase stability and structural defects in high-temperature Mo–Si–B alloys
journal, October 2008
- Sakidja, R.; Perepezko, J. H.; Kim, S.
- Acta Materialia, Vol. 56, Issue 18
Image Enhancement via Subimage Histogram Equalization Based on Mean and Variance
journal, January 2017
- Zhuang, Liyun; Guan, Yepeng
- Computational Intelligence and Neuroscience, Vol. 2017
Microstructure recognition using convolutional neural networks for prediction of ionic conductivity in ceramics
journal, December 2017
- Kondo, Ruho; Yamakawa, Shunsuke; Masuoka, Yumi
- Acta Materialia, Vol. 141
A systematic error in the determination of dislocation densities in thin films
journal, September 1961
- Ham, R. K.; Sharpe, N. G.
- Philosophical Magazine, Vol. 6, Issue 69
Automated defect analysis in electron microscopic images
journal, July 2018
- Li, Wei; Field, Kevin G.; Morgan, Dane
- npj Computational Materials, Vol. 4, Issue 1
Towards bend-contour-free dislocation imaging via diffraction contrast STEM
journal, October 2018
- Zhu, Yuanyuan; Ophus, Colin; Toloczko, Mychailo B.
- Ultramicroscopy, Vol. 193
A comparative study of texture measures with classification based on featured distributions
journal, January 1996
- Ojala, Timo; Pietikäinen, Matti; Harwood, David
- Pattern Recognition, Vol. 29, Issue 1
Fully Convolutional Networks for Semantic Segmentation
journal, April 2017
- Shelhamer, Evan; Long, Jonathan; Darrell, Trevor
- IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 4
Densely Connected Convolutional Networks
preprint, January 2016
- Huang, Gao; Liu, Zhuang; van der Maaten, Laurens
- arXiv
Image driven machine learning methods for microstructure recognition
journal, October 2016
- Chowdhury, Aritra; Kautz, Elizabeth; Yener, Bülent
- Computational Materials Science, Vol. 123
Dislocation movement through random arrays of obstacles
journal, October 1966
- Foreman, A. J. E.; Makin, M. J.
- Philosophical Magazine, Vol. 14, Issue 131
Importance-Aware Semantic Segmentation for Autonomous Vehicles
journal, January 2019
- Chen, Bike; Gong, Chen; Yang, Jian
- IEEE Transactions on Intelligent Transportation Systems, Vol. 20, Issue 1
Analytical Transmission Electron Microscopy
journal, August 2005
- Sigle, Wilfried
- Annual Review of Materials Research, Vol. 35, Issue 1
Dislocation Movement Through Random Arrays of Obstacles
journal, February 1967
- Foreman, A. J. E.; Makin, M. J.
- Canadian Journal of Physics, Vol. 45, Issue 2
Diffraction contrast STEM of dislocations: Imaging and simulations
journal, August 2011
- Phillips, P. J.; Brandes, M. C.; Mills, M. J.
- Ultramicroscopy, Vol. 111, Issue 9-10
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations
text, January 2018
- Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem
- arXiv
A computer vision approach for automated analysis and classification of microstructural image data
journal, December 2015
- DeCost, Brian L.; Holm, Elizabeth A.
- Computational Materials Science, Vol. 110
Fully convolutional networks for semantic segmentation
conference, June 2015
- Long, Jonathan; Shelhamer, Evan; Darrell, Trevor
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
The formation and interpretation of defect images from crystalline materials in a scanning transmission electron microscope
journal, January 1976
- Maher, Dennis M.; Joy, David C.
- Ultramicroscopy, Vol. 1, Issue 3-4
U-Net: Convolutional Networks for Biomedical Image Segmentation
book, November 2015
- Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas
- Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III
Densely Connected Convolutional Networks
conference, July 2017
- Huang, Gao; Liu, Zhuang; Maaten, Laurens van der
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Classification of corrosion defects in NiAl bronze through image analysis
journal, November 2010
- Pidaparti, Ramana M.; Aghazadeh, Babak Seyed; Whitfield, Angela
- Corrosion Science, Vol. 52, Issue 11
The formation and interpretation of defect images from crystalline materials in a scanning transmission electron microscope
journal, January 1976
- Maher, Dennis M.; Joy, David C.
- Ultramicroscopy, Vol. 1, Issue 3-4
Advanced Steel Microstructural Classification by Deep Learning Methods
journal, February 2018
- Azimi, Seyed Majid; Britz, Dominik; Engstler, Michael
- Scientific Reports, Vol. 8, Issue 1
Beyond Pixels: A Comprehensive Survey from Bottom-up to Semantic Image Segmentation and Cosegmentation
preprint, January 2015
- Zhu, Hongyuan; Meng, Fanman; Cai, Jianfei
- arXiv
A Method for Detection and Classification of Glass Defects in Low Resolution Images
conference, August 2011
- Zhao, Jie; Kong, Qing-Jie; Zhao, Xu
- Graphics (ICIG), 2011 Sixth International Conference on Image and Graphics
A review of semantic segmentation using deep neural networks
journal, November 2017
- Guo, Yanming; Liu, Yu; Georgiou, Theodoros
- International Journal of Multimedia Information Retrieval, Vol. 7, Issue 2
A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images
text, January 2018
- Madsen, Jacob; Liu, Pei; Kling, Jens
- arXiv
A Deep Learning Approach to Identify Local Structures in Atomic-Resolution Transmission Electron Microscopy Images
journal, July 2018
- Madsen, Jacob; Liu, Pei; Kling, Jens
- Advanced Theory and Simulations, Vol. 1, Issue 8
High throughput quantitative metallography for complex microstructures using deep learning: A case study in ultrahigh carbon steel
text, January 2018
- DeCost, Brian L.; Lei, Bo; Francis, Toby
- arXiv
A computer vision approach for automated analysis and classification of microstructural image data
journal, December 2015
- DeCost, Brian L.; Holm, Elizabeth A.
- Computational Materials Science, Vol. 110
The relationship between grain boundary structure, defect mobility and grain boundary sink efficiency
journal, March 2015
- Uberuaga, Blas Pedro; Vernon, Louis J.; Martinez, Enrique
- Scientific Reports, Vol. 5, Issue 1
Diffraction contrast STEM of dislocations: Imaging and simulations
journal, August 2011
- Phillips, P. J.; Brandes, M. C.; Mills, M. J.
- Ultramicroscopy, Vol. 111, Issue 9-10
Control and Use of Defects in Materials
journal, August 1998
- Szuromi, P.
- Science, Vol. 281, Issue 5379
AxonDeepSeg: automatic axon and myelin segmentation from microscopy data using convolutional neural networks
journal, February 2018
- Zaimi, Aldo; Wabartha, Maxime; Herman, Victor
- Scientific Reports, Vol. 8, Issue 1
A systematic error in the determination of dislocation densities in thin films
journal, September 1961
- Ham, R. K.; Sharpe, N. G.
- Philosophical Magazine, Vol. 6, Issue 69
Microstructure recognition using convolutional neural networks for prediction of ionic conductivity in ceramics
journal, December 2017
- Kondo, Ruho; Yamakawa, Shunsuke; Masuoka, Yumi
- Acta Materialia, Vol. 141
A Kinematical Theory of Diffraction Contrast of Electron Transmission Microscope Images of Dislocations and other Defects
journal, May 1960
- Hirsch, P. B.; Howie, A.; Whelan, M. J.
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 252, Issue 1017
A Kinematical Theory of Diffraction Contrast of Electron Transmission Microscope Images of Dislocations and other Defects
journal, May 1960
- Hirsch, P. B.; Howie, A.; Whelan, M. J.
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 252, Issue 1017
Phase stability and structural defects in high-temperature Mo–Si–B alloys
journal, October 2008
- Sakidja, R.; Perepezko, J. H.; Kim, S.
- Acta Materialia, Vol. 56, Issue 18
A comparative study of texture measures with classification based on featured distributions
journal, January 1996
- Ojala, Timo; Pietikäinen, Matti; Harwood, David
- Pattern Recognition, Vol. 29, Issue 1
Best practices for convolutional neural networks applied to visual document analysis
conference, January 2003
- Simard, P. Y.; Steinkraus, D.; Platt, J. C.
- Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings.
Image Enhancement via Subimage Histogram Equalization Based on Mean and Variance
journal, January 2017
- Zhuang, Liyun; Guan, Yepeng
- Computational Intelligence and Neuroscience, Vol. 2017
Disconnections and other defects associated with twin interfaces
journal, October 2016
- Hirth, J. P.; Wang, J.; Tomé, C. N.
- Progress in Materials Science, Vol. 83
Survey on semantic image segmentation techniques
conference, December 2017
- Sevak, Jay S.; Kapadia, Aerika D.; Chavda, Jaiminkumar B.
- 2017 International Conference on Intelligent Sustainable Systems (ICISS)
Analytical Transmission Electron Microscopy
datapaper, January 2009
- Wilfried, Sigle,
- Verlag der Technischen Universität Graz
Enhancing radiation tolerance by controlling defect mobility and migration pathways in multicomponent single-phase alloys
journal, December 2016
- Lu, Chenyang; Niu, Liangliang; Chen, Nanjun
- Nature Communications, Vol. 7, Issue 1
Advanced Steel Microstructural Classification by Deep Learning Methods
journal, February 2018
- Azimi, Seyed Majid; Britz, Dominik; Engstler, Michael
- Scientific Reports, Vol. 8, Issue 1
The relationship between grain boundary structure, defect mobility and grain boundary sink efficiency
journal, March 2015
- Uberuaga, Blas Pedro; Vernon, Louis J.; Martinez, Enrique
- Scientific Reports, Vol. 5, Issue 1
Works referencing / citing this record:
Learning-based Defect Recognition for Quasi-Periodic Microscope Images
text, January 2020
- Dennler, Nik; Foncubierta-Rodriguez, Antonio; Neupert, Titus
- arXiv