Automated defect analysis in electron microscopic images

Li, Wei; Field, Kevin G.; Morgan, Dane

doi:10.1038/s41524-018-0093-8

Automated defect analysis in electron microscopic images

Journal Article · Wed Jul 18 04:00:00 EDT 2018 · npj Computational Materials

DOI:https://doi.org/10.1038/s41524-018-0093-8· OSTI ID:1462882

^[1]; ^[2]; ^[1]

Univ. of Wisconsin, Madison, WI (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Electron microscopy and defect analysis are a cornerstone of materials science, as they offer detailed insights on the microstructure and performance of a wide range of materials and material systems. Building a robust and flexible platform for automated defect recognition and classification in electron microscopy will result in the completion of analysis orders of magnitude faster after images are recorded, or even online during image acquisition. Automated analysis has the potential to be significantly more efficient, accurate, and repeatable than human analysis, and it can scale with the increasingly important methods of automated data generation. Herein, an automated recognition tool is developed based on a computer vison–based approach; it sequentially applies a cascade object detector, convolutional neural network, and local image analysis methods. We demonstrate that the automated tool performs as well as or better than manual human detection in terms of recall and precision and achieves quantitative image/defect analysis metrics close to the human average. The proposed approach works for images of varying contrast, brightness, and magnification. Furthermore, these promising results suggest that this and similar approaches are worth exploring for detecting multiple defect types and have the potential to locate, classify, and measure quantitative features for a range of defect types, materials, and electron microscopic techniques.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1462882

Journal Information:: npj Computational Materials, Journal Name: npj Computational Materials Journal Issue: 1 Vol. 4; ISSN 2057-3960

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United States

Language:: English

References (44)

Visualizing and Understanding Convolutional Networks Zeiler, Matthew D.; Fergus, Rob Computer Vision – ECCV 2014 https://doi.org/10.1007/978-3-319-10590-1_53	book	January 2014
Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks Cireşan, Dan C.; Giusti, Alessandro; Gambardella, Luca M. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013 https://doi.org/10.1007/978-3-642-40763-5_51	book	January 2013
Structural materials for fission & fusion energy Zinkle, Steven J.; Busby, Jeremy T. Materials Today, Vol. 12, Issue 11 https://doi.org/10.1016/S1369-7021(09)70294-9	journal	November 2009
Exploring the microstructure manifold: Image texture representations applied to ultrahigh carbon steel microstructures DeCost, Brian L.; Francis, Toby; Holm, Elizabeth A. Acta Materialia, Vol. 133 https://doi.org/10.1016/j.actamat.2017.05.014	journal	July 2017
Image driven machine learning methods for microstructure recognition Chowdhury, Aritra; Kautz, Elizabeth; Yener, Bülent Computational Materials Science, Vol. 123 https://doi.org/10.1016/j.commatsci.2016.05.034	journal	October 2016
TEM characterization of dislocation loops in irradiated bcc Fe-based steels Yao, B.; Edwards, D. J.; Kurtz, R. J. Journal of Nuclear Materials, Vol. 434, Issue 1-3 https://doi.org/10.1016/j.jnucmat.2012.12.002	journal	March 2013
Radiation tolerance of neutron-irradiated model Fe–Cr–Al alloys Field, Kevin G.; Hu, Xunxiang; Littrell, Kenneth C. Journal of Nuclear Materials, Vol. 465 https://doi.org/10.1016/j.jnucmat.2015.06.023	journal	October 2015
Heterogeneous dislocation loop formation near grain boundaries in a neutron-irradiated commercial FeCrAl alloy Field, Kevin G.; Briggs, Samuel A.; Hu, Xunxiang Journal of Nuclear Materials, Vol. 483 https://doi.org/10.1016/j.jnucmat.2016.10.050	journal	January 2017
Dislocation loop formation in model FeCrAl alloys after neutron irradiation below 1 dpa Field, Kevin G.; Briggs, Samuel A.; Sridharan, Kumar Journal of Nuclear Materials, Vol. 495 https://doi.org/10.1016/j.jnucmat.2017.07.061	journal	November 2017
Diffraction contrast STEM of dislocations: Imaging and simulations Phillips, P. J.; Brandes, M. C.; Mills, M. J. Ultramicroscopy, Vol. 111, Issue 9-10 https://doi.org/10.1016/j.ultramic.2011.07.001	journal	August 2011
A deep convolutional neural network to analyze position averaged convergent beam electron diffraction patterns Xu, W.; LeBeau, J. M. Ultramicroscopy, Vol. 188 https://doi.org/10.1016/j.ultramic.2018.03.004	journal	May 2018
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem ACS Nano, Vol. 11, Issue 12 https://doi.org/10.1021/acsnano.7b07504	journal	October 2016
Robust Real-Time Face Detection Viola, Paul; Jones, Michael J. International Journal of Computer Vision, Vol. 57, Issue 2 https://doi.org/10.1023/B:VISI.0000013087.49260.fb	journal	May 2004
Deep learning LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey Nature, Vol. 521, Issue 7553 https://doi.org/10.1038/nature14539	journal	May 2015
Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography Jesse, S.; Chi, M.; Belianinov, A. Scientific Reports, Vol. 6, Issue 1 https://doi.org/10.1038/srep26348	journal	May 2016
Inferring low-dimensional microstructure representations using convolutional neural networks Lubbers, Nicholas; Lookman, Turab; Barros, Kipton Physical Review E, Vol. 96, Issue 5 https://doi.org/10.1103/PhysRevE.96.052111	journal	November 2017
Watersheds in digital spaces: an efficient algorithm based on immersion simulations Vincent, L.; Soille, P. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 13, Issue 6 https://doi.org/10.1109/34.87344	journal	June 1991
Shape matching and object recognition using shape contexts Belongie, S.; Malik, J.; Puzicha, J. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, Issue 4 https://doi.org/10.1109/34.993558	journal	April 2002
Rapid object detection using a boosted cascade of simple features Viola, P.; Jones, M. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 https://doi.org/10.1109/CVPR.2001.990517	conference	January 2001
Histograms of Oriented Gradients for Human Detection Dalal, N.; Triggs, B. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) https://doi.org/10.1109/CVPR.2005.177	conference	January 2005
Fully convolutional networks for semantic segmentation Long, Jonathan; Shelhamer, Evan; Darrell, Trevor 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2015.7298965	conference	June 2015
Rethinking the Inception Architecture for Computer Vision Szegedy, Christian; Vanhoucke, Vincent; Ioffe, Sergey 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.308	conference	June 2016
Face Detection with the Faster R-CNN Jiang, Huaizu; Learned-Miller, Erik 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) https://doi.org/10.1109/FG.2017.82	conference	May 2017
Object recognition from local scale-invariant features Lowe, D. G. Proceedings of the Seventh IEEE International Conference on Computer Vision https://doi.org/10.1109/ICCV.1999.790410	conference	January 1999
Mask R-CNN He, Kaiming; Gkioxari, Georgia; Dollar, Piotr 2017 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2017.322	conference	October 2017
Better matching with fewer features: The selection of useful features in large database recognition problems Turcot, Panu; Lowe, David G. 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops https://doi.org/10.1109/ICCVW.2009.5457541	conference	September 2009
A Method for Detection and Classification of Glass Defects in Low Resolution Images Zhao, Jie; Kong, Qing-Jie; Zhao, Xu Graphics (ICIG), 2011 Sixth International Conference on Image and Graphics https://doi.org/10.1109/ICIG.2011.187	conference	August 2011
A practical implementation of face detection by using Matlab cascade object detector Alionte, Elena; Lazar, Corneliu 2015 19th International Conference on System Theory, Control and Computing (ICSTCC) https://doi.org/10.1109/ICSTCC.2015.7321390	conference	October 2015
A general review of human face detection including a study of neural networks and Haar feature-based cascade classifier in face detection Sharifara, Ali; Mohd Rahim, Mohd Shafry; Anisi, Yasaman 2014 International Symposium on Biometrics and Security Technologies (ISBAST) https://doi.org/10.1109/ISBAST.2014.7013097	conference	August 2014
Face Description with Local Binary Patterns: Application to Face Recognition Ahonen, T.; Hadid, A.; Pietikainen, M. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28, Issue 12 https://doi.org/10.1109/TPAMI.2006.244	journal	December 2006
Object Detection with Discriminatively Trained Part-Based Models Felzenszwalb, P. F.; Girshick, R. B.; McAllester, D. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, Issue 9 https://doi.org/10.1109/TPAMI.2009.167	journal	September 2010
Robust real-time face detection Viola, P.; Jones, M. Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 https://doi.org/10.1109/iccv.2001.937709	conference	January 2001
Fully Convolutional Networks for Semantic Segmentation Shelhamer, Evan; Long, Jonathan; Darrell, Trevor IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 4 https://doi.org/10.1109/tpami.2016.2572683	journal	April 2017
Mask R-CNN He, Kaiming; Gkioxari, Georgia; Dollar, Piotr IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, Issue 2 https://doi.org/10.1109/tpami.2018.2844175	journal	February 2020
Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets Belianinov, Alex; Vasudevan, Rama; Strelcov, Evgheni Advanced Structural and Chemical Imaging, Vol. 1, Issue 1 https://doi.org/10.1186/s40679-015-0006-6	journal	May 2015
Automated Detection and Segmentation of Synaptic Contacts in Nearly Isotropic Serial Electron Microscopy Images Kreshuk, Anna; Straehle, Christoph N.; Sommer, Christoph PLoS ONE, Vol. 6, Issue 10 https://doi.org/10.1371/journal.pone.0024899	journal	October 2011
Characterization of irradiation defect structures and densities by transmission electron microscopy Kirk, Marquis; Yi, Xiaoou; Jenkins, Michael Journal of Materials Research, Vol. 30, Issue 9 https://doi.org/10.1557/jmr.2015.19	journal	February 2015
Application of STEM characterization for investigating radiation effects in BCC Fe-based alloys Parish, Chad M.; Field, Kevin G.; Certain, Alicia G. Journal of Materials Research, Vol. 30, Issue 9 https://doi.org/10.1557/jmr.2015.32	journal	April 2015
Detection of Open Loop Defects in STEM Images of Irradiation-Damaged Alloys – Source Code for Detection and Image Dataset Li, Wei; Field, Kevin G.; Morgan, Dane Materials Data Facility https://doi.org/10.18126/m2692z	dataset	January 2018
Rethinking the Inception Architecture for Computer Vision Szegedy, Christian; Vanhoucke, Vincent; Ioffe, Sergey arXiv https://doi.org/10.48550/arxiv.1512.00567	preprint	January 2015
Fully Convolutional Networks for Semantic Segmentation Shelhamer, Evan; Long, Jonathan; Darrell, Trevor arXiv https://doi.org/10.48550/arxiv.1605.06211	preprint	January 2016
A Deep Convolutional Neural Network to Analyze Position Averaged Convergent Beam Electron Diffraction Patterns Xu, Weizong; LeBeau, James M. arXiv https://doi.org/10.48550/arxiv.1708.00855	text	January 2017
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem arXiv https://doi.org/10.48550/arxiv.1801.05860	text	January 2018
Deep Learning Ganatra, Sanket Zenodo https://doi.org/10.5281/zenodo.1189359	text	January 2018

Cited By (4)

Interplay Between Grain Boundaries and Radiation Damage Barr, Christopher M.; El-Atwani, Osman; Kaoumi, Djamel JOM, Vol. 71, Issue 4 https://doi.org/10.1007/s11837-019-03386-y	journal	March 2019
Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels Roberts, Graham; Haile, Simon Y.; Sainju, Rajat Scientific Reports, Vol. 9, Issue 1 https://doi.org/10.1038/s41598-019-49105-0	journal	September 2019
Machine-learning-assisted screening of pure-silica zeolites for effective removal of linear siloxanes and derivatives Lin, Shiru; Wang, Yekun; Zhao, Yinghe Journal of Materials Chemistry A, Vol. 8, Issue 6 https://doi.org/10.1039/c9ta11909d	journal	January 2020
Data-driven approach for synchrotron X-ray Laue microdiffraction scan analysis Song, Yintao; Tamura, Nobumichi; Zhang, Chenbo Acta Crystallographica Section A Foundations and Advances, Vol. 75, Issue 6 https://doi.org/10.1107/s2053273319012804	journal	October 2019

Similar Records

Detection of defects in atomic-resolution images of materials using cycle analysis

Journal Article · Mon Mar 30 00:00:00 EDT 2020 · Advanced Structural and Chemical Imaging · OSTI ID:1649061

Multi defect detection and analysis of electron microscopy images with deep learning

Journal Article · Mon Aug 02 00:00:00 EDT 2021 · Computational Materials Science · OSTI ID:1845761

Systematic Approach for Validation of X-Ray Automatic Defect Recognition Systems

Journal Article · Wed Mar 21 00:00:00 EDT 2007 · AIP Conference Proceedings · OSTI ID:21054958

Related Subjects

47 OTHER INSTRUMENTATION

Automated defect analysis in electron microscopic images

Citation Formats

References (44)

Cited By (4)

Linked Research (1)

Similar Records

Related Subjects