CoordNet: Data Generation and Visualization Generation for Time-Varying Volumes via a Coordinate-Based Neural Network

Han, Jun; Wang, Chaoli

doi:10.1109/tvcg.2022.3197203

CoordNet: Data Generation and Visualization Generation for Time-Varying Volumes via a Coordinate-Based Neural Network

Journal Article · Mon Aug 08 04:00:00 EDT 2022 · IEEE Transactions on Visualization and Computer Graphics

DOI:https://doi.org/10.1109/tvcg.2022.3197203· OSTI ID:2345684

^[1]; ^[2]

Chinese University of Hong Kong, Guangdong (China); University of Notre Dame
University of Notre Dame, IN (United States)

Although deep learning has demonstrated its capability in solving diverse scientific visualization problems, it still lacks generalization power across different tasks. To address this challenge, we propose CoordNet, a single coordinate-based framework that tackles various tasks relevant to time-varying volumetric data visualization without modifying the network architecture. The core idea of our approach is to decompose diverse task inputs and outputs into a unified representation (i.e., coordinates and values) and learn a function from coordinates to their corresponding values. We achieve this goal using a residual block-based implicit neural representation architecture with periodic activation functions. We evaluate CoordNet on data generation (i.e., temporal super-resolution and spatial super-resolution) and visualization generation (i.e., view synthesis and ambient occlusion prediction) tasks using time-varying volumetric data sets of various characteristics. Here, the experimental results indicate that CoordNet achieves better quantitative and qualitative results than the state-of-the-art approaches across all the evaluated tasks. Source code and pre-trained models are available at https://github.com/stevenhan1991/CoordNet.

View Accepted Manuscript (DOE)

Research Organization:: University of Notre Dame, IN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0023145

OSTI ID:: 2345684

Journal Information:: IEEE Transactions on Visualization and Computer Graphics, Journal Name: IEEE Transactions on Visualization and Computer Graphics Journal Issue: 12 Vol. 29; ISSN 1077-2626

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (29)

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis Mildenhall, Ben; Srinivasan, Pratul P.; Tancik, Matthew Computer Vision – ECCV 2020 https://doi.org/10.1007/978-3-030-58452-8_24	book	November 2020
VCNet: A generative model for volume completion Han, Jun; Wang, Chaoli Visual Informatics, Vol. 6, Issue 2 https://doi.org/10.1016/j.visinf.2022.04.004	journal	June 2022
Ionization Front Instabilities in Primordial H ii Regions Whalen, Daniel; Norman, Michael L. The Astrophysical Journal, Vol. 673, Issue 2 https://doi.org/10.1086/524400	journal	February 2008
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network Shi, Wenzhe; Caballero, Jose; Huszar, Ferenc 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.207	conference	June 2016
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric Zhang, Richard; Isola, Phillip; Efros, Alexei A. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00068	conference	June 2018
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation Jiang, Huaizu; Sun, Deqing; Jampani, Varan 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00938	conference	June 2018
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision Niemeyer, Michael; Mescheder, Lars; Oechsle, Michael 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00356	conference	June 2020
12-in-1: Multi-Task Vision and Language Representation Learning Lu, Jiasen; Goswami, Vedanuj; Rohrbach, Marcus 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.01045	conference	June 2020
Learned Initializations for Optimizing Coordinate-Based Neural Representations Tancik, Matthew; Mildenhall, Ben; Wang, Terrance 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR46437.2021.00287	conference	June 2021
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis Chan, Eric R.; Monteiro, Marco; Kellnhofer, Petr 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR46437.2021.00574	conference	June 2021
UniT: Multimodal Multitask Learning with a Unified Transformer Hu, Ronghang; Singh, Amanpreet 2021 IEEE/CVF International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV48922.2021.00147	conference	October 2021
Importance-Driven Time-Varying Data Visualization No authors listed IEEE Transactions on Visualization and Computer Graphics, Vol. 14, Issue 6 https://doi.org/10.1109/TVCG.2008.140	journal	November 2008
A Generative Model for Volume Rendering Berger, Matthew; Li, Jixian; Levine, Joshua A. IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 4 https://doi.org/10.1109/TVCG.2018.2816059	journal	April 2019
TSR-TVD: Temporal Super-Resolution for Time-Varying Data Analysis and Visualization Han, Jun; Wang, Chaoli IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2019.2934255	journal	January 2019
InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations He, Wenbin; Wang, Junpeng; Guo, Hanqi IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2019.2934312	journal	January 2019
Vector Field Topology of Time-Dependent Flows in a Steady Reference Frame Rojo, Irene Baeza; Gunther, Tobias IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2019.2934375	journal	January 2019
A Fluid Flow Data Set for Machine Learning and its Application to Neural Flow Map Interpolation Jakob, Jakob; Gross, Markus; Gunther, Tobias IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 2 https://doi.org/10.1109/TVCG.2020.3028947	journal	February 2021
Deep Volumetric Ambient Occlusion Engel, Dominik; Ropinski, Timo IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 2 https://doi.org/10.1109/TVCG.2020.3030344	journal	February 2021
V2V: A Deep Learning Approach to Variable-to-Variable Selection and Translation for Multivariate Time-Varying Data Han, Jun; Zheng, Hao; Xing, Yunhao IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 2 https://doi.org/10.1109/TVCG.2020.3030346	journal	February 2021
SSR-TVD: Spatial Super-Resolution for Time-Varying Data Analysis and Visualization Han, Jun; Wang, Chaoli IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2020.3032123	journal	January 2020
Learning Adaptive Sampling and Reconstruction for Volume Visualization Weiss, Sebastian; Isk, Mustafa; Thies, Justus IEEE Transactions on Visualization and Computer Graphics, Vol. 28, Issue 7 https://doi.org/10.1109/TVCG.2020.3039340	journal	July 2022
STNet: An End-to-End Generative Framework for Synthesizing Spatiotemporal Super-Resolution Volumes Han, Jun; Zheng, Hao; Chen, Danny Z. IEEE Transactions on Visualization and Computer Graphics, Vol. 28, Issue 1 https://doi.org/10.1109/TVCG.2021.3114815	journal	January 2022
DL4SciVis: A State-of-the-Art Survey on Deep Learning for Scientific Visualization Wang, Chaoli; Han, Jun IEEE Transactions on Visualization and Computer Graphics, Vol. 29, Issue 8 https://doi.org/10.1109/tvcg.2022.3167896	journal	August 2023
Compressive Neural Representations of Volumetric Scalar Fields Lu, Y.; Jiang, K.; Levine, J. A. Computer Graphics Forum, Vol. 40, Issue 3 https://doi.org/10.1111/cgf.14295	journal	June 2021
Volume upscaling with convolutional neural networks Zhou, Zhenglei; Hou, Yule; Wang, Qirui Proceedings of the Computer Graphics International Conference https://doi.org/10.1145/3095140.3095178	conference	June 2017
CNNs Based Viewpoint Estimation for Volume Visualization Shi, Neng; Tao, Yubo ACM Transactions on Intelligent Systems and Technology, Vol. 10, Issue 3 https://doi.org/10.1145/3309993	journal	April 2019
Experimental and Numerical Study of the Turbulence Characteristics of Airflow around a Research Vessel Popinet, Stéphane; Smith, Murray; Stevens, Craig Journal of Atmospheric and Oceanic Technology, Vol. 21, Issue 10 https://doi.org/10.1175/1520-0426(2004)021<1575:eansot>2.0.co;2	journal	October 2004
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks Hashimoto, Kazuma; xiong, caiming; Tsuruoka, Yoshimasa Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing https://doi.org/10.18653/v1/D17-1206	conference	January 2017

Similar Records

STSR-INR: Spatiotemporal super-resolution for multivariate time-varying volumetric data via implicit neural representation

Journal Article · Mon Jan 15 23:00:00 EST 2024 · Computers and Graphics · OSTI ID:2281315

GMT: A deep learning approach to generalized multivariate translation for scientific data analysis and visualization

Journal Article · Tue Apr 11 00:00:00 EDT 2023 · Computers and Graphics · OSTI ID:1972200

NeRVI: Compressive neural representation of visualization images for communicating volume visualization results

Journal Article · Tue Aug 22 00:00:00 EDT 2023 · Computers and Graphics · OSTI ID:1995970

Related Subjects

97 MATHEMATICS AND COMPUTING
Volume visualization
data generation
implicit neural representation
visualization generation

CoordNet: Data Generation and Visualization Generation for Time-Varying Volumes via a Coordinate-Based Neural Network

Citation Formats

References (29)

Similar Records

Related Subjects