A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials
Abstract
In computed TEM tomography, image segmentation represents one of the most basic tasks with implications not only for 3D volume visualization, but more importantly for quantitative 3D analysis. In case of large and complex 3D data sets, segmentation can be an extremely difficult and laborious task, and thus has been one of the biggest hurdles for comprehensive 3D analysis. Heterogeneous catalysts have complex surface and bulk structures, and often sparse distribution of catalytic particles with relatively poor intrinsic contrast, which possess a unique challenge for image segmentation, including the current state-of-the-art deep learning methods. To tackle this problem, we apply a deep learning-based approach for the multi-class semantic segmentation of a γ-Alumina/Pt catalytic material in a class imbalance situation. Specifically, we used the weighted focal loss as a loss function and attached it to the U-Net’s fully convolutional network architecture. We assessed the accuracy of our results using Dice similarity coefficient (DSC), recall, precision, and Hausdorff distance (HD) metrics on the overlap between the ground-truth and predicted segmentations. Our adopted U-Net model with the weighted focal loss function achieved an average DSC score of 0.96 ± 0.003 in the γ-Alumina support material and 0.84 ± 0.03 in the Pt NPsmore »
- Authors:
-
- The Ohio State Univ., Columbus, OH (United States)
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Publication Date:
- Research Org.:
- Pacific Northwest National Laboratory (PNNL), Richland, WA (United States). Environmental Molecular Sciences Laboratory (EMSL)
- Sponsoring Org.:
- USDOE Office of Science (SC), Biological and Environmental Research (BER); USDOE Office of Science (SC), Basic Energy Sciences (BES). Chemical Sciences, Geosciences & Biosciences Division
- OSTI Identifier:
- 1895125
- Report Number(s):
- PNNL-SA-174381
Journal ID: ISSN 2045-2322
- Grant/Contract Number:
- AC05-76RL01830
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Scientific Reports
- Additional Journal Information:
- Journal Volume: 12; Journal Issue: 1; Journal ID: ISSN 2045-2322
- Publisher:
- Nature Publishing Group
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 36 MATERIALS SCIENCE; Materials for energy and catalysis; Transmission electron microscopy
Citation Formats
Genc, Arda, Kovarik, Libor, and Fraser, Hamish L. A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials. United States: N. p., 2022.
Web. doi:10.1038/s41598-022-16429-3.
Genc, Arda, Kovarik, Libor, & Fraser, Hamish L. A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials. United States. https://doi.org/10.1038/s41598-022-16429-3
Genc, Arda, Kovarik, Libor, and Fraser, Hamish L. Wed .
"A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials". United States. https://doi.org/10.1038/s41598-022-16429-3. https://www.osti.gov/servlets/purl/1895125.
@article{osti_1895125,
title = {A deep learning approach for semantic segmentation of unbalanced data in electron tomography of catalytic materials},
author = {Genc, Arda and Kovarik, Libor and Fraser, Hamish L.},
abstractNote = {In computed TEM tomography, image segmentation represents one of the most basic tasks with implications not only for 3D volume visualization, but more importantly for quantitative 3D analysis. In case of large and complex 3D data sets, segmentation can be an extremely difficult and laborious task, and thus has been one of the biggest hurdles for comprehensive 3D analysis. Heterogeneous catalysts have complex surface and bulk structures, and often sparse distribution of catalytic particles with relatively poor intrinsic contrast, which possess a unique challenge for image segmentation, including the current state-of-the-art deep learning methods. To tackle this problem, we apply a deep learning-based approach for the multi-class semantic segmentation of a γ-Alumina/Pt catalytic material in a class imbalance situation. Specifically, we used the weighted focal loss as a loss function and attached it to the U-Net’s fully convolutional network architecture. We assessed the accuracy of our results using Dice similarity coefficient (DSC), recall, precision, and Hausdorff distance (HD) metrics on the overlap between the ground-truth and predicted segmentations. Our adopted U-Net model with the weighted focal loss function achieved an average DSC score of 0.96 ± 0.003 in the γ-Alumina support material and 0.84 ± 0.03 in the Pt NPs segmentation tasks. We report an average boundary-overlap error of less than 2 nm at the 90th percentile of HD for γ-Alumina and Pt NPs segmentations. The complex surface morphology of γ-Alumina and its relation to the Pt NPs were visualized in 3D by the deep learning-assisted automatic segmentation of a large data set of high-angle annular dark-field (HAADF) scanning transmission electron microscopy (STEM) tomography reconstructions.},
doi = {10.1038/s41598-022-16429-3},
journal = {Scientific Reports},
number = 1,
volume = 12,
place = {United States},
year = {Wed Sep 28 00:00:00 EDT 2022},
month = {Wed Sep 28 00:00:00 EDT 2022}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Fully convolutional networks for semantic segmentation
conference, June 2015
- Long, Jonathan; Shelhamer, Evan; Darrell, Trevor
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Object matching algorithms using robust Hausdorff distance measures
journal, March 1999
- Dong-Gyu Sim,
- IEEE Transactions on Image Processing, Vol. 8, Issue 3
Embedded Nanostructures Revealed in Three Dimensions
journal, September 2005
- Arslan, I.
- Science, Vol. 309, Issue 5744
DeepEM3D: approaching human-level performance on 3D anisotropic EM image segmentation
journal, March 2017
- Zeng, Tao; Wu, Bian; Ji, Shuiwang
- Bioinformatics, Vol. 33, Issue 16
U-Net: Convolutional Networks for Biomedical Image Segmentation
book, November 2015
- Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas
- Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III
TomoPy: a framework for the analysis of synchrotron tomographic data
journal, August 2014
- Gürsoy, Dogˇa; De Carlo, Francesco; Xiao, Xianghui
- Journal of Synchrotron Radiation, Vol. 21, Issue 5
Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels
journal, September 2019
- Roberts, Graham; Haile, Simon Y.; Sainju, Rajat
- Scientific Reports, Vol. 9, Issue 1
Focal Loss for Dense Object Detection
journal, February 2020
- Lin, Tsung-Yi; Goyal, Priya; Girshick, Ross
- IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, Issue 2
Deep learning
journal, May 2015
- LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
- Nature, Vol. 521, Issue 7553
Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images
journal, July 2020
- Horwath, James P.; Zakharov, Dmitri N.; Mégret, Rémi
- npj Computational Materials, Vol. 6, Issue 1
High Throughput Quantitative Metallography for Complex Microstructures Using Deep Learning: A Case Study in Ultrahigh Carbon Steel
journal, February 2019
- DeCost, Brian L.; Lei, Bo; Francis, Toby
- Microscopy and Microanalysis, Vol. 25, Issue 1
Hausdorff Distance with Outliers and Noise Resilience Capabilities
journal, June 2021
- Maiseli, Baraka Jacob
- SN Computer Science, Vol. 2, Issue 5
Precise Identification and Characterization of Catalytically Active Sites on the Surface of γ‐Alumina**
journal, June 2021
- Khivantsev, Konstantin; Jaegers, Nicholas R.; Kwak, Ja‐Hun
- Angewandte Chemie, Vol. 133, Issue 32
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
conference, October 2016
- Milletari, Fausto; Navab, Nassir; Ahmadi, Seyed-Ahmad
- 2016 Fourth International Conference on 3D Vision (3DV)
Fully Convolutional Architectures for Multiclass Segmentation in Chest Radiographs
journal, August 2018
- Novikov, Alexey A.; Lenis, Dimitrios; Major, David
- IEEE Transactions on Medical Imaging, Vol. 37, Issue 8
Rapid and flexible segmentation of electron microscopy data using few-shot machine learning
journal, November 2021
- Akers, Sarah; Kautz, Elizabeth; Trevino-Gavito, Andrea
- npj Computational Materials, Vol. 7, Issue 1
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
journal, August 2015
- Taha, Abdel Aziz; Hanbury, Allan
- BMC Medical Imaging, Vol. 15, Issue 1
3D electron microscopy in the physical sciences: the development of Z-contrast and EFTEM tomography
journal, September 2003
- Midgley, P. A.; Weyland, M.
- Ultramicroscopy, Vol. 96, Issue 3-4
3D-TEM investigation of the nanostructure of a δ-Al 2 O 3 catalyst support decorated with Pd nanoparticles
journal, January 2012
- Roiban, Lucian; Sorbier, Loïc; Pichon, Christophe
- Nanoscale, Vol. 4, Issue 3
Weighted Focal Loss: An Effective Loss Function to Overcome Unbalance Problem of Chest X-ray14
journal, October 2018
- Qin, Ruoxi; Qiao, Kai; Wang, Linyuan
- IOP Conference Series: Materials Science and Engineering, Vol. 428
Tomography and High-Resolution Electron Microscopy Study of Surfaces and Porosity in a Plate-like γ-Al 2 O 3
journal, December 2012
- Kovarik, Libor; Genc, Arda; Wang, Chongmin
- The Journal of Physical Chemistry C, Vol. 117, Issue 1
Deep learning STEM-EDX tomography of nanocrystals
journal, February 2021
- Han, Yoseob; Jang, Jaeduck; Cha, Eunju
- Nature Machine Intelligence, Vol. 3, Issue 3
2D & 3D in situ study of the calcination of Pd nanocatalysts supported on delta-Alumina in an Environmental Transmission Electron Microscope
journal, August 2019
- Epicier, Thierry; Koneti, Siddardha; Avenier, Priscilla
- Catalysis Today, Vol. 334
Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique
journal, May 2016
- Greenspan, Hayit; van Ginneken, Bram; Summers, Ronald M.
- IEEE Transactions on Medical Imaging, Vol. 35, Issue 5
Loss Weightings for Improving Imbalanced Brain Structure Segmentation Using Fully Convolutional Networks
journal, July 2021
- Sugino, Takaaki; Kawase, Toshihiro; Onogi, Shinya
- Healthcare, Vol. 9, Issue 8
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
conference, December 2015
- He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing
- 2015 IEEE International Conference on Computer Vision (ICCV)
Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation
journal, January 2022
- Yeung, Michael; Sala, Evis; Schönlieb, Carola-Bibiane
- Computerized Medical Imaging and Graphics, Vol. 95
A survey of loss functions for semantic segmentation
conference, October 2020
- Jadon, Shruti
- 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB)
Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning
journal, November 2021
- Greenwald, Noah F.; Miller, Geneva; Moen, Erick
- Nature Biotechnology, Vol. 40, Issue 4
Maximum Likelihood Reconstruction for Emission Tomography
journal, October 1982
- Shepp, L. A.; Vardi, Y.
- IEEE Transactions on Medical Imaging, Vol. 1, Issue 2
Electron Tomography of Nanoparticle Catalysts on Porous Supports: A New Technique Based on Rutherford Scattering
journal, August 2001
- Weyland, Matthew; Midgley, Paul A.; Thomas, John Meurig
- The Journal of Physical Chemistry B, Vol. 105, Issue 33
Comparing images using the Hausdorff distance
journal, January 1993
- Huttenlocher, D. P.; Klanderman, G. A.; Rucklidge, W. J.
- IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 15, Issue 9
Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations
book, January 2017
- Sudre, Carole H.; Li, Wenqi; Vercauteren, Tom
- Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support
0.7 Å Resolution Electron Tomography Enabled by Deep‐Learning‐Aided Information Recovery
journal, September 2020
- Wang, Chunyang; Ding, Guanglei; Liu, Yitong
- Advanced Intelligent Systems, Vol. 2, Issue 12
ParaView: An End-User Tool for Large-Data Visualization
book, January 2005
- Ahrens, James; Geveci, Berk; Law, Charles
- Visualization Handbook, p. 717-731
Rapid alignment of nanotomography data using joint iterative reconstruction and reprojection
journal, September 2017
- Gürsoy, Doğa; Hong, Young P.; He, Kuan
- Scientific Reports, Vol. 7, Issue 1
Integration of TomoPy and the ASTRA toolbox for advanced processing and reconstruction of tomographic synchrotron data
journal, April 2016
- Pelt, Daniël M.; Gürsoy, Dogˇa; Palenstijn, Willem Jan
- Journal of Synchrotron Radiation, Vol. 23, Issue 3
Nanoscale scanning transmission electron tomography
journal, September 2006
- Midgley, P. A.; Weyland, M.; Yates, T. J. V.
- Journal of Microscopy, Vol. 223, Issue 3
Z-Contrast tomography: a technique in three-dimensional nanostructural analysis based on Rutherford scattering
journal, January 2001
- Midgley, Paul A.; Weyland, Matthew; Thomas, John Meurig
- Chemical Communications, Issue 10