GMT: A deep learning approach to generalized multivariate translation for scientific data analysis and visualization

Yao, Siyuan; Han, Jun; Wang, Chaoli

doi:10.1016/j.cag.2023.04.002

GMT: A deep learning approach to generalized multivariate translation for scientific data analysis and visualization

Journal Article · Tue Apr 11 00:00:00 EDT 2023 · Computers and Graphics

DOI:https://doi.org/10.1016/j.cag.2023.04.002· OSTI ID:1972200

^[1]; ^[2]; ^[3]

Univ. of Notre Dame, IN (United States); University of Notre Dame
Chinese Univ. of Hong Kong, Guangdong (China)
Univ. of Notre Dame, IN (United States)

In scientific visualization, despite the significant advances of deep learning for data generation, researchers have not thoroughly investigated the issue of data translation. We present a new deep learning approach called generalized multivariate translation (GMT) for multivariate time-varying data analysis and visualization. Like V2V, GMT assumes a preprocessing step that selects suitable variables for translation. However, unlike V2V, which only handles one-to-one variable translation during training and inference, GMT enables one-to-many and many-to-many variable translation in the same framework. We leverage the recent StarGAN design from multi-domain image-to-image translation to achieve this generalization capability. We experiment with different loss functions and injection strategies to explore the best choices and leverage pre-training for performance improvement. We compare GMT with other state-of-the-art methods (i.e., Pix2Pix, V2V, StarGAN). Furthermore, the results demonstrate the overall advantage of GMT in translation quality and generalization ability.

View Accepted Manuscript (DOE)

Research Organization:: Univ. of Notre Dame, IN (United States)

Sponsoring Organization:: U.S. National Science Foundation; USDOE

Grant/Contract Number:: SC0023145

OSTI ID:: 1972200

Alternate ID(s):: OSTI ID: 1971952

Journal Information:: Computers and Graphics, Journal Name: Computers and Graphics Vol. 112; ISSN 0097-8493

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (28)

Multimodal Unsupervised Image-to-Image Translation Huang, Xun; Liu, Ming-Yu; Belongie, Serge Computer Vision – ECCV 2018 https://doi.org/10.1007/978-3-030-01219-9_11	book	January 2018
kCBAC-Net: Deeply Supervised Complete Bipartite Networks with Asymmetric Convolutions for Medical Image Segmentation Gu, Pengfei; Zheng, Hao; Zhang, Yizhe Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 https://doi.org/10.1007/978-3-030-87193-2_32	book	January 2021
U-Net: Convolutional Networks for Biomedical Image Segmentation Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III https://doi.org/10.1007/978-3-319-24574-4_28	book	November 2015
TSR-VFD: Generating temporal super-resolution for unsteady vector field data Han, Jun; Wang, Chaoli Computers & Graphics, Vol. 103 https://doi.org/10.1016/j.cag.2022.02.001	journal	April 2022
Mid-mantle heterogeneities and iron spin transition in the lower mantle: Implications for mid-mantle slab stagnation Shahnas, M. H.; Yuen, D. A.; Pysklywec, R. N. Earth and Planetary Science Letters, Vol. 458 https://doi.org/10.1016/j.epsl.2016.10.052	journal	January 2017
Scalar mixing in direct numerical simulations of temporally evolving plane jet flames with skeletal CO/H2 kinetics Hawkes, Evatt R.; Sankaran, Ramanan; Sutherland, James C. Proceedings of the Combustion Institute, Vol. 31, Issue 1 https://doi.org/10.1016/j.proci.2006.08.079	journal	January 2007
Ionization Front Instabilities in Primordial H ii Regions Whalen, Daniel; Norman, Michael L. The Astrophysical Journal, Vol. 673, Issue 2 https://doi.org/10.1086/524400	journal	February 2008
Image-to-Image Translation with Conditional Adversarial Networks Isola, Phillip; Zhu, Jun-Yan; Zhou, Tinghui 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.632	conference	July 2017
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric Zhang, Richard; Isola, Phillip; Efros, Alexei A. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00068	conference	June 2018
Controllable Video Generation with Sparse Trajectories Hao, Zekun; Huang, Xun; Belongie, Serge 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00819	conference	June 2018
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation Choi, Yunjey; Choi, Minje; Kim, Munyoung 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2018.00916	conference	June 2018
High-Resolution Image Synthesis with Latent Diffusion Models Rombach, Robin; Blattmann, Andreas; Lorenz, Dominik 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.01042	conference	June 2022
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2015 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2015.123	conference	December 2015
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization Huang, Xun; Belongie, Serge 2017 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2017.167	conference	October 2017
STSRNet: Deep Joint Space–Time Super-Resolution for Vector Field Visualization An, Yifei; Shen, Han-Wei; Shan, Guihua IEEE Computer Graphics and Applications, Vol. 41, Issue 6 https://doi.org/10.1109/MCG.2021.3097555	journal	November 2021
SSR-VFD: Spatial Super-Resolution for Vector Field Data Analysis and Visualization Guo, Li; Ye, Shaojie; Han, Jun 2020 IEEE Pacific Visualization Symposium (PacificVis) https://doi.org/10.1109/PacificVis48177.2020.8737	conference	June 2020
Scalar2Vec: Translating Scalar Fields to Vector Fields via Deep Learning Gu, Pengfei; Han, Jun; Chen, Danny Z. 2022 IEEE 15th Pacific Visualization Symposium (PacificVis) https://doi.org/10.1109/PacificVis53943.2022.00012	conference	April 2022
Image Quality Assessment: From Error Visibility to Structural Similarity Wang, Z.; Bovik, A. C.; Sheikh, H. R. IEEE Transactions on Image Processing, Vol. 13, Issue 4 https://doi.org/10.1109/TIP.2003.819861	journal	April 2004
TTHRESH: Tensor Compression for Multidimensional Visual Data Ballester-Ripoll, Rafael; Lindstrom, Peter; Pajarola, Renato IEEE Transactions on Visualization and Computer Graphics, Vol. 26, Issue 9 https://doi.org/10.1109/TVCG.2019.2904063	journal	September 2020
A Fluid Flow Data Set for Machine Learning and its Application to Neural Flow Map Interpolation Jakob, Jakob; Gross, Markus; Gunther, Tobias IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 2 https://doi.org/10.1109/TVCG.2020.3028947	journal	February 2021
Deep Volumetric Ambient Occlusion Engel, Dominik; Ropinski, Timo IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 2 https://doi.org/10.1109/TVCG.2020.3030344	journal	February 2021
V2V: A Deep Learning Approach to Variable-to-Variable Selection and Translation for Multivariate Time-Varying Data Han, Jun; Zheng, Hao; Xing, Yunhao IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 2 https://doi.org/10.1109/TVCG.2020.3030346	journal	February 2021
STNet: An End-to-End Generative Framework for Synthesizing Spatiotemporal Super-Resolution Volumes Han, Jun; Zheng, Hao; Chen, Danny Z. IEEE Transactions on Visualization and Computer Graphics, Vol. 28, Issue 1 https://doi.org/10.1109/TVCG.2021.3114815	journal	January 2022
Deep Fluids: A Generative Network for Parameterized Fluid Simulations Kim, Byungsoo; Azevedo, Vinicius C.; Thuerey, Nils Computer Graphics Forum, Vol. 38, Issue 2 https://doi.org/10.1111/cgf.13619	journal	May 2019
Compressive Neural Representations of Volumetric Scalar Fields Lu, Y.; Jiang, K.; Levine, J. A. Computer Graphics Forum, Vol. 40, Issue 3 https://doi.org/10.1111/cgf.14295	journal	June 2021
Volume upscaling with convolutional neural networks Zhou, Zhenglei; Hou, Yule; Wang, Qirui Proceedings of the Computer Graphics International Conference https://doi.org/10.1145/3095140.3095178	conference	June 2017
Learning meaningful controls for fluids Chu, Mengyu; Thuerey, Nils; Seidel, Hans-Peter ACM Transactions on Graphics, Vol. 40, Issue 4 https://doi.org/10.1145/3450626.3459845	journal	July 2021
Experimental and Numerical Study of the Turbulence Characteristics of Airflow around a Research Vessel Popinet, Stéphane; Smith, Murray; Stevens, Craig Journal of Atmospheric and Oceanic Technology, Vol. 21, Issue 10 https://doi.org/10.1175/1520-0426(2004)021<1575:EANSOT>2.0.CO;2	journal	October 2004

Similar Records

STSR-INR: Spatiotemporal super-resolution for multivariate time-varying volumetric data via implicit neural representation

Journal Article · Mon Jan 15 19:00:00 EST 2024 · Computers and Graphics · OSTI ID:2281315

NeRVI: Compressive neural representation of visualization images for communicating volume visualization results

Journal Article · Mon Aug 21 20:00:00 EDT 2023 · Computers and Graphics · OSTI ID:1995970

Related Subjects

97 MATHEMATICS AND COMPUTING
Deep learning
Multivariate time-varying data
Scientific visualization
Variable-to-variable translation

GMT: A deep learning approach to generalized multivariate translation for scientific data analysis and visualization

Citation Formats

References (28)

Similar Records

Related Subjects