Interactive Visual Study of Multiple Attributes Learning Model of X-Ray Scattering Images

Huang, Xinyi; Jamonnak, Suphanut; Zhao, Ye; Wang, Boyu; Hoai, Minh; Yager, Kevin; Xu, Wei

doi:10.1109/tvcg.2020.3030384

Interactive Visual Study of Multiple Attributes Learning Model of X-Ray Scattering Images

Journal Article · 26 October 2020 · IEEE Transactions on Visualization and Computer Graphics

DOI:https://doi.org/10.1109/tvcg.2020.3030384· OSTI ID:1782541

Huang, Xinyi ^[1]; Jamonnak, Suphanut ^[1]; Zhao, Ye ^[1]; Wang, Boyu ^[2]; Hoai, Minh ^[2]; Yager, Kevin ^[3]; Xu, Wei ^[3]

Kent State Univ., Kent, OH (United States)
Stony Brook Univ., NY (United States)
Brookhaven National Lab. (BNL), Upton, NY (United States)

Existing interactive visualization tools for deep learning are mostly applied to the training, debugging, and refinement of neural network models working on natural images. However, visual analytics tools are lacking for the specific application of x-ray image classification with multiple structural attributes. In this paper, we present an interactive system for domain scientists to visually study the multiple attributes learning models applied to x-ray scattering images. It allows domain scientists to interactively explore this important type of scientific images in embedded spaces that are defined on the model prediction output, the actual labels, and the discovered feature space of neural networks. Users are allowed to flexibly select instance images, their clusters, and compare them regarding the specified visual representation of attributes. The exploration is guided by the manifestation of model performance related to mutual relationships among attributes, which often affect the learning accuracy and effectiveness. The system thus supports domain scientists to improve the training dataset and model, find questionable attributes labels, and identify outlier images or spurious data clusters. Case studies and scientists feedback demonstrate its functionalities and usefulness.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (SC-21)

Grant/Contract Number:: SC0012704

OSTI ID:: 1782541

Report Number(s):: BNL--221352-2021-JAAM

Journal Information:: IEEE Transactions on Visualization and Computer Graphics, Journal Name: IEEE Transactions on Visualization and Computer Graphics Journal Issue: 2 Vol. 27; ISSN 1077-2626

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (33)

Silhouettes: A graphical aid to the interpretation and validation of cluster analysis Rousseeuw, Peter J. Journal of Computational and Applied Mathematics, Vol. 20 https://doi.org/10.1016/0377-0427(87)90125-7	journal	November 1987
A Dendrite Method for Cluster Analysis Calinski, T.; Harabasz, J. Communications in Statistics - Simulation and Computation, Vol. 3, Issue 1 https://doi.org/10.1080/03610917408548446	journal	January 1974
Classification of crystal structure using a convolutional neural network Park, Woon Bae; Chung, Jiyong; Jung, Jaeyoung IUCrJ, Vol. 4, Issue 4 https://doi.org/10.1107/s205225251700714x	journal	June 2017
Dataset of Synthetic X-ray Scattering Images for Classification Using Deep Learning Yager, Kevin G.; Lhermitte, Julien; Yu, Dantong Materials Data Facility https://doi.org/10.18126/m2z30z	dataset	January 2017
Embedding Projector: Interactive Visualization and Interpretation of Embeddings Smilkov, Daniel; Thorat, Nikhil; Nicholson, Charles arXiv https://doi.org/10.48550/arxiv.1611.05469	preprint	January 2016
Direct-Manipulation Visualization of Deep Networks Smilkov, Daniel; Carter, Shan; Sculley, D. arXiv https://doi.org/10.48550/arxiv.1708.03788	preprint	January 2017
Synergy, redundancy, and multivariate information measures: an experimentalist’s perspective Timme, Nicholas; Alford, Wesley; Flecker, Benjamin Journal of Computational Neuroscience, Vol. 36, Issue 2 https://doi.org/10.1007/s10827-013-0458-4	journal	July 2013
A dendrite method for cluster analysis Calinski, T.; Harabasz, J. Communications in Statistics - Theory and Methods, Vol. 3, Issue 1 https://doi.org/10.1080/03610927408827101	journal	January 1974
Periodic lattices of arbitrary nano-objects: modeling and applications for self-assembled systems Yager, Kevin G.; Zhang, Yugang; Lu, Fang Journal of Applied Crystallography, Vol. 47, Issue 1 https://doi.org/10.1107/S160057671302832X	journal	December 2013
BraggNet: integrating Bragg peaks using neural networks Sullivan, Brendan; Archibald, Rick; Azadmanesh, Jahaun Journal of Applied Crystallography, Vol. 52, Issue 4 https://doi.org/10.1107/S1600576719008665	journal	July 2019
Classification of crystal structure using a convolutional neural network Park, Woon Bae; Chung, Jiyong; Jung, Jaeyoung IUCrJ, Vol. 4, Issue 4 https://doi.org/10.1107/S205225251700714X	journal	June 2017
Learning Deep Features for Discriminative Localization Zhou, Bolei; Khosla, Aditya; Lapedriza, Agata 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.319	conference	June 2016
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
YOLO9000: Better, Faster, Stronger Redmon, Joseph; Farhadi, Ali 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.690	conference	July 2017
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization Selvaraju, Ramprasaath R.; Cogswell, Michael; Das, Abhishek 2017 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2017.74	conference	October 2017
Visual Analytics for Explainable Deep Learning Choo, Jaegul; Liu, Shixia IEEE Computer Graphics and Applications, Vol. 38, Issue 4 https://doi.org/10.1109/MCG.2018.042731661	journal	July 2018
Object Detection With Deep Learning: A Review Zhao, Zhong-Qiu; Zheng, Peng; Xu, Shou-Tao IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, Issue 11 https://doi.org/10.1109/TNNLS.2018.2876865	journal	November 2019
A Cluster Separation Measure Davies, David L.; Bouldin, Donald W. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-1, Issue 2 https://doi.org/10.1109/TPAMI.1979.4766909	journal	April 1979
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation Badrinarayanan, Vijay; Kendall, Alex; Cipolla, Roberto IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 12 https://doi.org/10.1109/TPAMI.2016.2644615	journal	December 2017
Towards Better Analysis of Deep Convolutional Neural Networks Liu, Mengchen; Shi, Jiaxin; Li, Zhen IEEE Transactions on Visualization and Computer Graphics, Vol. 23, Issue 1 https://doi.org/10.1109/TVCG.2016.2598831	journal	January 2017
Visualizing the Hidden Activity of Artificial Neural Networks Rauber, Paulo E.; Fadel, Samuel G.; Falcao, Alexandre X. IEEE Transactions on Visualization and Computer Graphics, Vol. 23, Issue 1 https://doi.org/10.1109/TVCG.2016.2598838	journal	January 2017
DeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks Pezzotti, Nicola; Hollt, Thomas; Van Gemert, Jan IEEE Transactions on Visualization and Computer Graphics, Vol. 24, Issue 1 https://doi.org/10.1109/TVCG.2017.2744358	journal	January 2018
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models Kahng, Minsuk; Andrews, Pierre Y.; Kalro, Aditya IEEE Transactions on Visualization and Computer Graphics, Vol. 24, Issue 1 https://doi.org/10.1109/TVCG.2017.2744718	journal	January 2018
Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow Wongsuphasawat, Kanit; Smilkov, Daniel; Wexler, James IEEE Transactions on Visualization and Computer Graphics, Vol. 24, Issue 1 https://doi.org/10.1109/TVCG.2017.2744878	journal	January 2018
Analyzing the Training Processes of Deep Generative Models Liu, Mengchen; Shi, Jiaxin; Cao, Kelei IEEE Transactions on Visualization and Computer Graphics, Vol. 24, Issue 1 https://doi.org/10.1109/TVCG.2017.2744938	journal	January 2018
GANViz: A Visual Analytics Approach to Understand the Adversarial Game Wang, Junpeng; Gou, Liang; Yang, Hao IEEE Transactions on Visualization and Computer Graphics, Vol. 24, Issue 6 https://doi.org/10.1109/TVCG.2018.2816223	journal	June 2018
GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation Kahng, Minsuk; Thorat, Nikhil; Chau, Duen Horng Polo IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 1 https://doi.org/10.1109/TVCG.2018.2864500	journal	January 2019
Materials discovery: Fine-grained classification of X-ray scattering images Hadi Kiapour, M.; Yager, Kevin; Berg, Alexander C. 2014 IEEE Winter Conference on Applications of Computer Vision (WACV) https://doi.org/10.1109/WACV.2014.6836004	conference	March 2014
X-Ray Scattering Image Classification Using Deep Learning Wang, Boyu; Yager, Kevin; Yu, Dantong 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) https://doi.org/10.1109/WACV.2017.83	conference	March 2017
Development of an instrument measuring user satisfaction of the human-computer interface Chin, J. P.; Diehl, V. A.; Norman, L. K. Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '88 https://doi.org/10.1145/57167.57203	conference	January 1988
Convolutional neural networks for grazing incidence x-ray scattering patterns: thin film structure identification Liu, Shuai; Melton, Charles N.; Venkatakrishnan, Singanallur MRS Communications, Vol. 9, Issue 02 https://doi.org/10.1557/mrc.2019.26	journal	March 2019
A causal framework for explaining the predictions of black-box sequence-to-sequence models Alvarez-Melis, David; Jaakkola, Tommi Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing https://doi.org/10.18653/v1/D17-1042	conference	January 2017
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier Ribeiro, Marco; Singh, Sameer; Guestrin, Carlos Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations https://doi.org/10.18653/v1/N16-3020	conference	January 2016

Similar Records

NeRVI: Compressive neural representation of visualization images for communicating volume visualization results

Journal Article · 2023 · Computers and Graphics · OSTI ID:1995970

Visualization for Classification in Deep Neural Networks

Conference · 2017 · OSTI ID:1407764

DLSIA: Deep Learning for Scientific Image Analysis

Journal Article · 2024 · Journal of Applied Crystallography (Online) · OSTI ID:2328638

Related Subjects

97 MATHEMATICS AND COMPUTING

Interactive Visual Study of Multiple Attributes Learning Model of X-Ray Scattering Images

Citation Formats

References (33)

Similar Records

Related Subjects