Multi defect detection and analysis of electron microscopy images with deep learning

Shen, Mingren; Li, Guanzhao; Wu, Dongxia; Liu, Yuhan; Greaves, Jacob R. C.; Hao, Wei; Krakauer, Nathaniel J.; Krudy, Leah; Perez, Jacob; Sreenivasan, Varun; Sanchez, Bryan; Torres-Velázquez, Oigimer; Li, Wei; Field, Kevin G.; Morgan, Dane

doi:10.1016/j.commatsci.2021.110576

Multi defect detection and analysis of electron microscopy images with deep learning

Journal Article · Mon Aug 02 00:00:00 EDT 2021 · Computational Materials Science

DOI:https://doi.org/10.1016/j.commatsci.2021.110576· OSTI ID:1845761

Shen, Mingren ^[1]; ^[1]; Wu, Dongxia ^[1]; ^[1]; Greaves, Jacob R. C. ^[1]; Hao, Wei ^[1]; Krakauer, Nathaniel J. ^[1]; Krudy, Leah ^[2]; Perez, Jacob ^[1]; ^[1]; ^[3]; Torres-Velázquez, Oigimer ^[3]; Li, Wei ^[1]; ^[4]; ^[1]

Univ. of Wisconsin, Madison, WI (United States)
Hope College, Holland, MI (United States)
Univ. of Puerto Rico at Mayagüez (Puerto Rico)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States); Univ. of Michigan, Ann Arbor, MI (United States)

Electron microscopy is widely used to explore defects in crystal structures, but human detecting of defects is often time-consuming, error-prone, and unreliable, and is not scalable to large numbers of images or real-time analysis. In this work, we discuss the application of machine learning approaches to find the location and geometry of different defect clusters in irradiated steels. We show that a deep learning based Faster R-CNN analysis system has a performance comparable to human analysis with relatively small training data sets. Furthermore, this study proves the promising ability to apply deep learning to assist the development of automated microscopy data analysis even when multiple features are present and paves the way for fast, scalable, and reliable analysis systems for massive amounts of modern electron microscopy data.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; USDOE Office of Nuclear Energy (NE)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1845761

Alternate ID(s):: OSTI ID: 1811600

Journal Information:: Computational Materials Science, Journal Name: Computational Materials Science Vol. 199; ISSN 0927-0256

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (38)

Computer Vision and Machine Learning for Autonomous Characterization of AM Powder Feedstocks DeCost, Brian L.; Jain, Harshvardhan; Rollett, Anthony D. JOM, Vol. 69, Issue 3 https://doi.org/10.1007/s11837-016-2226-1	journal	December 2016
Structural materials for fission & fusion energy Zinkle, Steven J.; Busby, Jeremy T. Materials Today, Vol. 12, Issue 11 https://doi.org/10.1016/S1369-7021(09)70294-9	journal	November 2009
Exploring the microstructure manifold: Image texture representations applied to ultrahigh carbon steel microstructures DeCost, Brian L.; Francis, Toby; Holm, Elizabeth A. Acta Materialia, Vol. 133 https://doi.org/10.1016/j.actamat.2017.05.014	journal	July 2017
A computer vision approach for automated analysis and classification of microstructural image data DeCost, Brian L.; Holm, Elizabeth A. Computational Materials Science, Vol. 110 https://doi.org/10.1016/j.commatsci.2015.08.011	journal	December 2015
Image driven machine learning methods for microstructure recognition Chowdhury, Aritra; Kautz, Elizabeth; Yener, Bülent Computational Materials Science, Vol. 123 https://doi.org/10.1016/j.commatsci.2016.05.034	journal	October 2016
Characterizing powder materials using keypoint-based computer vision methods DeCost, Brian L.; Holm, Elizabeth A. Computational Materials Science, Vol. 126 https://doi.org/10.1016/j.commatsci.2016.08.038	journal	January 2017
Advanced microstructure classification by data mining methods Gola, Jessica; Britz, Dominik; Staudt, Thorsten Computational Materials Science, Vol. 148 https://doi.org/10.1016/j.commatsci.2018.03.004	journal	June 2018
A large dataset of synthetic SEM images of powder materials and their ground truth 3D structures DeCost, Brian L.; Holm, Elizabeth A. Data in Brief, Vol. 9 https://doi.org/10.1016/j.dib.2016.10.011	journal	December 2016
TEM characterization of dislocation loops in irradiated bcc Fe-based steels Yao, B.; Edwards, D. J.; Kurtz, R. J. Journal of Nuclear Materials, Vol. 434, Issue 1-3 https://doi.org/10.1016/j.jnucmat.2012.12.002	journal	March 2013
Radiation tolerance of neutron-irradiated model Fe–Cr–Al alloys Field, Kevin G.; Hu, Xunxiang; Littrell, Kenneth C. Journal of Nuclear Materials, Vol. 465 https://doi.org/10.1016/j.jnucmat.2015.06.023	journal	October 2015
Heterogeneous dislocation loop formation near grain boundaries in a neutron-irradiated commercial FeCrAl alloy Field, Kevin G.; Briggs, Samuel A.; Hu, Xunxiang Journal of Nuclear Materials, Vol. 483 https://doi.org/10.1016/j.jnucmat.2016.10.050	journal	January 2017
Dislocation loop formation in model FeCrAl alloys after neutron irradiation below 1 dpa Field, Kevin G.; Briggs, Samuel A.; Sridharan, Kumar Journal of Nuclear Materials, Vol. 495 https://doi.org/10.1016/j.jnucmat.2017.07.061	journal	November 2017
Automatic segmentation of inorganic nanoparticles in BF TEM micrographs Groom, D. J.; Yu, K.; Rasouli, S. Ultramicroscopy, Vol. 194 https://doi.org/10.1016/j.ultramic.2018.06.002	journal	November 2018
Automated Detection of Helium Bubbles in Irradiated X-750 Anderson, Chris M.; Klein, Jacob; Rajakumar, Heygaan Ultramicroscopy, Vol. 217 https://doi.org/10.1016/j.ultramic.2020.113068	journal	October 2020
Knowledge Extraction from Atomically Resolved Images Vlcek, Lukas; Maksov, Artem; Pan, Minghu ACS Nano, Vol. 11, Issue 10 https://doi.org/10.1021/acsnano.7b05036	journal	October 2017
Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations Ziatdinov, Maxim; Dyck, Ondrej; Maksov, Artem ACS Nano, Vol. 11, Issue 12 https://doi.org/10.1021/acsnano.7b07504	journal	October 2016
Deep learning LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey Nature, Vol. 521, Issue 7553 https://doi.org/10.1038/nature14539	journal	May 2015
Big–deep–smart data in imaging for guiding materials design Kalinin, Sergei V.; Sumpter, Bobby G.; Archibald, Richard K. Nature Materials, Vol. 14, Issue 10 https://doi.org/10.1038/nmat4395	journal	September 2015
Fiji: an open-source platform for biological-image analysis Schindelin, Johannes; Arganda-Carreras, Ignacio; Frise, Erwin Nature Methods, Vol. 9, Issue 7 https://doi.org/10.1038/nmeth.2019	journal	June 2012
NIH Image to ImageJ: 25 years of image analysis Schneider, Caroline A.; Rasband, Wayne S.; Eliceiri, Kevin W. Nature Methods, Vol. 9, Issue 7 https://doi.org/10.1038/nmeth.2089	journal	June 2012
Automated defect analysis in electron microscopic images Li, Wei; Field, Kevin G.; Morgan, Dane npj Computational Materials, Vol. 4, Issue 1 https://doi.org/10.1038/s41524-018-0093-8	journal	July 2018
An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis Chen, Po-Hsuan Cameron; Gadepalli, Krishna; MacDonald, Robert Nature Medicine, Vol. 25, Issue 9 https://doi.org/10.1038/s41591-019-0539-7	journal	August 2019
The Hessian Blob Algorithm: Precise Particle Detection in Atomic Force Microscopy Imagery Marsh, Brendan P.; Chada, Nagaraju; Sanganna Gari, Raghavendar Reddy Scientific Reports, Vol. 8, Issue 1 https://doi.org/10.1038/s41598-018-19379-x	journal	January 2018
Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels Roberts, Graham; Haile, Simon Y.; Sainju, Rajat Scientific Reports, Vol. 9, Issue 1 https://doi.org/10.1038/s41598-019-49105-0	journal	September 2019
Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography Jesse, S.; Chi, M.; Belianinov, A. Scientific Reports, Vol. 6, Issue 1 https://doi.org/10.1038/srep26348	journal	May 2016
Rapid object detection using a boosted cascade of simple features Viola, P.; Jones, M. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 https://doi.org/10.1109/CVPR.2001.990517	conference	January 2001
Image processing for materials characterization: Issues, challenges and opportunities Duval, L.; Moreaud, M.; Couprie, C. 2014 IEEE International Conference on Image Processing (ICIP) https://doi.org/10.1109/ICIP.2014.7025985	conference	October 2014
A Survey on Transfer Learning Pan, Sinno Jialin; Yang, Qiang IEEE Transactions on Knowledge and Data Engineering, Vol. 22, Issue 10 https://doi.org/10.1109/TKDE.2009.191	journal	October 2010
Object Detection With Deep Learning: A Review Zhao, Zhong-Qiu; Zheng, Peng; Xu, Shou-Tao IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, Issue 11 https://doi.org/10.1109/TNNLS.2018.2876865	journal	November 2019
Face Description with Local Binary Patterns: Application to Face Recognition Ahonen, T.; Hadid, A.; Pietikainen, M. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28, Issue 12 https://doi.org/10.1109/TPAMI.2006.244	journal	December 2006
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Ren, Shaoqing; He, Kaiming; Girshick, Ross IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 6 https://doi.org/10.1109/TPAMI.2016.2577031	journal	June 2017
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation Badrinarayanan, Vijay; Kendall, Alex; Cipolla, Roberto IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 12 https://doi.org/10.1109/TPAMI.2016.2644615	journal	December 2017
ChainerCV: a Library for Deep Learning in Computer Vision Niitani, Yusuke; Ogawa, Toru; Saito, Shunta MM '17: ACM Multimedia Conference, Proceedings of the 25th ACM international conference on Multimedia https://doi.org/10.1145/3123266.3129395	conference	October 2017
A survey on Image Data Augmentation for Deep Learning Shorten, Connor; Khoshgoftaar, Taghi M. Journal of Big Data, Vol. 6, Issue 1 https://doi.org/10.1186/s40537-019-0197-0	journal	July 2019
Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography Somnath, Suhas; Jesse, Stephen; Belianinov, Alex Oak Ridge Leadership Computing Facility; Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States) https://doi.org/10.13139/olcf/1463599	dataset	January 2018
Application of STEM characterization for investigating radiation effects in BCC Fe-based alloys Parish, Chad M.; Field, Kevin G.; Certain, Alicia G. Journal of Materials Research, Vol. 30, Issue 9 https://doi.org/10.1557/jmr.2015.32	journal	April 2015
Automating material image analysis for material discovery Park, Chiwoo; Ding, Yu MRS Communications, Vol. 9, Issue 02 https://doi.org/10.1557/mrc.2019.48	journal	April 2019
An Overview of Watershed Algorithm Implementations in Open Source Libraries Kornilov, Anton; Safonov, Ilia Journal of Imaging, Vol. 4, Issue 10 https://doi.org/10.3390/jimaging4100123	journal	October 2018

Similar Records

Automated correlative segmentation of large Transmission X-ray Microscopy (TXM) tomograms using deep learning

Journal Article · Wed Aug 15 00:00:00 EDT 2018 · Materials Characterization · OSTI ID:22805091

Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels

Journal Article · Tue Sep 03 20:00:00 EDT 2019 · Scientific Reports · OSTI ID:1559992

Deep Learning Analysis of Polaritonic Wave Images

Journal Article · Thu Oct 28 20:00:00 EDT 2021 · ACS Nano · OSTI ID:1866114

Related Subjects

36 MATERIALS SCIENCE
Deep learning
Defect detection and analysis
Electron microscopy
Faster R-CNN

Multi defect detection and analysis of electron microscopy images with deep learning

Citation Formats

References (38)

Similar Records

Related Subjects