STSR-INR: Spatiotemporal super-resolution for multivariate time-varying volumetric data via implicit neural representation

Tang, Kaiyuan; Wang, Chaoli

doi:10.1016/j.cag.2024.01.001

Title: STSR-INR: Spatiotemporal super-resolution for multivariate time-varying volumetric data via implicit neural representation

Journal Article · 15 January 2024 · Computers and Graphics

DOI: https://doi.org/10.1016/j.cag.2024.01.001 · OSTI ID:2281315

^[1]; Wang, Chaoli ^[2]

University of Notre Dame, IN (United States); University of Notre Dame
University of Notre Dame, IN (United States)

Implicit neural representation (INR) has surfaced as a promising direction for solving different scientific visualization tasks due to its continuous representation and flexible input and output settings. We present STSR-INR, an INR solution for generating simultaneous spatiotemporal super-resolution for multivariate time-varying volumetric data. Inheriting the benefits of the INR-based approach, STSR-INR supports unsupervised learning and permits data upscaling with arbitrary spatial and temporal scale factors. Unlike existing GAN- or INR-based super-resolution methods, STSR-INR focuses on tackling variables or ensembles and enabling joint training across datasets of various spatiotemporal resolutions. Here we achieve this capability via a variable embedding scheme that learns latent vectors for different variables. In conjunction with a modulated structure in the network design, we employ a variational auto-decoder to optimize the learnable latent vectors to enable latent-space interpolation. To combat the slow training of INR, we leverage a multi-head strategy to improve training and inference speed with significant speedup. We demonstrate the effectiveness of STSR-INR with multiple scalar field datasets and compare it with conventional tricubic+linear interpolation and state-of-the-art deep-learning-based solutions (STNet and CoordNet).

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: University of Notre Dame, IN (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0023145

OSTI ID:: 2281315

Journal Information:: Computers and Graphics, Journal Name: Computers and Graphics Vol. 119; ISSN 0097-8493

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (32)

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis Mildenhall, Ben; Srinivasan, Pratul P.; Tancik, Matthew Computer Vision – ECCV 2020 https://doi.org/10.1007/978-3-030-58452-8_24	book	November 2020
Scalar mixing in direct numerical simulations of temporally evolving plane jet flames with skeletal CO/H2 kinetics Hawkes, Evatt R.; Sankaran, Ramanan; Sutherland, James C. Proceedings of the Combustion Institute, Vol. 31, Issue 1 https://doi.org/10.1016/j.proci.2006.08.079	journal	January 2007
Ionization Front Instabilities in Primordial H ii Regions Whalen, Daniel; Norman, Michael L. The Astrophysical Journal, Vol. 673, Issue 2 https://doi.org/10.1086/524400	journal	February 2008
Tracking and visualizing turbulent 3D features Silver, D. IEEE Transactions on Visualization and Computer Graphics, Vol. 3, Issue 2 https://doi.org/10.1109/2945.597796	journal	January 1997
Phase-based frame interpolation for video Meyer, Simone; Wang, Oliver; Zimmer, Henning 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2015.7298747	conference	June 2015
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric Zhang, Richard; Isola, Phillip; Efros, Alexei A. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00068	conference	June 2018
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation Jiang, Huaizu; Sun, Deqing; Jampani, Varan 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00938	conference	June 2018
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation Park, Jeong Joon; Florence, Peter; Straub, Julian 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00025	conference	June 2019
Feedback Network for Image Super-Resolution Li, Zhen; Yang, Jinglei; Liu, Zheng 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00399	conference	June 2019
VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution Chen, Zeyuan; Chen, Yinbo; Liu, Jingwen 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.00209	conference	June 2022
Neural 3D Video Synthesis from Multi-view Video Li, Tianye; Slavcheva, Mira; Zollhoefer, Michael 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.00544	conference	June 2022
A Structured Dictionary Perspective on Implicit Neural Representations Yuce, Gizem; Ortiz-Jimenez, Guillermo; Besbinar, Beril 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.01863	conference	June 2022
Multi-Head Relu Implicit Neural Representation Networks Aftab, Arya; Morsali, Alireza; Ghaemmaghami, Shahrokh ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP43922.2022.9747352	conference	May 2022
Video Frame Interpolation via Adaptive Separable Convolution Niklaus, Simon; Mai, Long; Liu, Feng 2017 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2017.37	conference	October 2017
Modulated Periodic Activations for Generalizable Local Functional Representations Mehta, Ishit; Gharbi, Michael; Barnes, Connelly 2021 IEEE/CVF International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV48922.2021.01395	conference	October 2021
SwinIR: Image Restoration Using Swin Transformer Liang, Jingyun; Cao, Jiezhang; Sun, Guolei 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) https://doi.org/10.1109/ICCVW54120.2021.00210	conference	October 2021
An Arbitrary Scale Super-Resolution Approach for 3D MR Images via Implicit Neural Representation Wu, Qing; Li, Yuwei; Sun, Yawen IEEE Journal of Biomedical and Health Informatics, Vol. 27, Issue 2 https://doi.org/10.1109/JBHI.2022.3223106	journal	February 2023
Reconstructing Unsteady Flow Data From Representative Streamlines via Diffusion and Deep-Learning-Based Denoising Gu, Pengfei; Han, Jun; Chen, Danny Z. IEEE Computer Graphics and Applications, Vol. 41, Issue 6 https://doi.org/10.1109/MCG.2021.3089627	journal	November 2021
STSRNet: Deep Joint Space–Time Super-Resolution for Vector Field Visualization An, Yifei; Shen, Han-Wei; Shan, Guihua IEEE Computer Graphics and Applications, Vol. 41, Issue 6 https://doi.org/10.1109/MCG.2021.3097555	journal	November 2021
Scalar2Vec: Translating Scalar Fields to Vector Fields via Deep Learning Gu, Pengfei; Han, Jun; Chen, Danny Z. 2022 IEEE 15th Pacific Visualization Symposium (PacificVis) https://doi.org/10.1109/PacificVis53943.2022.00012	conference	April 2022
Image Super-Resolution Using Deep Convolutional Networks Dong, Chao; Loy, Chen Change; He, Kaiming IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, Issue 2 https://doi.org/10.1109/TPAMI.2015.2439281	journal	February 2016
A Statistical Approach to Volume Data Quality Assessment IEEE Transactions on Visualization and Computer Graphics, Vol. 14, Issue 3 https://doi.org/10.1109/TVCG.2007.70628	journal	May 2008
V2V: A Deep Learning Approach to Variable-to-Variable Selection and Translation for Multivariate Time-Varying Data Han, Jun; Zheng, Hao; Xing, Yunhao IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 2 https://doi.org/10.1109/TVCG.2020.3030346	journal	February 2021
STNet: An End-to-End Generative Framework for Synthesizing Spatiotemporal Super-Resolution Volumes Han, Jun; Zheng, Hao; Chen, Danny Z. IEEE Transactions on Visualization and Computer Graphics, Vol. 28, Issue 1 https://doi.org/10.1109/TVCG.2021.3114815	journal	January 2022
DL4SciVis: A State-of-the-Art Survey on Deep Learning for Scientific Visualization Wang, Chaoli; Han, Jun IEEE Transactions on Visualization and Computer Graphics, Vol. 29, Issue 8 https://doi.org/10.1109/TVCG.2022.3167896	journal	August 2023
CoordNet: Data Generation and Visualization Generation for Time-Varying Volumes via a Coordinate-Based Neural Network Han, Jun; Wang, Chaoli IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2022.3197203	journal	January 2022
Interactive Volume Visualization Via Multi-Resolution Hash Encoding Based Neural Representation Wu, Qi; Bauer, David; Doyle, Michael J. IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2023.3293121	journal	January 2023
Compressive Neural Representations of Volumetric Scalar Fields Lu, Y.; Jiang, K.; Levine, J. A. Computer Graphics Forum, Vol. 40, Issue 3 https://doi.org/10.1111/cgf.14295	journal	June 2021
Fast Neural Representations for Direct Volume Rendering Weiss, S.; Hermüller, P.; Westermann, R. Computer Graphics Forum, Vol. 41, Issue 6 https://doi.org/10.1111/cgf.14578	journal	June 2022
Volume upscaling with convolutional neural networks Zhou, Zhenglei; Hou, Yule; Wang, Qirui Proceedings of the Computer Graphics International Conference https://doi.org/10.1145/3095140.3095178	conference	June 2017
Experimental and Numerical Study of the Turbulence Characteristics of Airflow around a Research Vessel Popinet, Stéphane; Smith, Murray; Stevens, Craig Journal of Atmospheric and Oceanic Technology, Vol. 21, Issue 10 https://doi.org/10.1175/1520-0426(2004)021<1575:EANSOT>2.0.CO;2	journal	October 2004
Exploratory Lagrangian-Based Particle Tracing Using deep Learning Han, Mengjiao; Sane, Sudhanshu; Johnson, Chris R. Journal of Flow Visualization and Image Processing, Vol. 29, Issue 3 https://doi.org/10.1615/JFlowVisImageProc.2022041197	journal	January 2022

Similar Records

CoordNet: Data Generation and Visualization Generation for Time-Varying Volumes via a Coordinate-Based Neural Network

Journal Article · 2022 · IEEE Transactions on Visualization and Computer Graphics · OSTI ID:2345684

Han, Jun; Wang, Chaoli

Learning nuclear cross sections across the chart of nuclides with graph neural networks

Journal Article · 2025 · Physical Review. C · OSTI ID:2999046

Choi, Hongjun; Mitra, Sinjini; Brodksy, Jason; +6 more

Machine Learning Approach for Spatiotemporal Multivariate Optimization of Environmental Monitoring Sensor Locations

Journal Article · 2024 · Artificial Intelligence for the Earth Systems · OSTI ID:2476587

Siddiquee, Masudur R.; Meray, Aurelien O.; Xu, Zexuan; +6 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Implicit neural representation
Multivariate time-varying data
Spatiotemporal super-resolution

Title: STSR-INR: Spatiotemporal super-resolution for multivariate time-varying volumetric data via implicit neural representation

Citation Formats

References (32)

Similar Records

Related Subjects