NeRVI: Compressive neural representation of visualization images for communicating volume visualization results

Gu, Pengfei; Chen, Danny Z.; Wang, Chaoli

doi:10.1016/j.cag.2023.08.024

NeRVI: Compressive neural representation of visualization images for communicating volume visualization results

Journal Article · Tue Aug 22 04:00:00 EDT 2023 · Computers and Graphics

DOI:https://doi.org/10.1016/j.cag.2023.08.024· OSTI ID:1995970

^[1]; ^[2]; ^[2]

University of Notre Dame, IN (United States); University of Notre Dame
University of Notre Dame, IN (United States)

We present NeRVI, a new deep-learning approach that compresses a large collection of visualization images generated from time-varying data for communicating volume visualization results. Based on an image-based implicit neural representation, our approach represents tens of thousands of high-resolution rendering images parametrized by different parameters via a hybrid model of multilayer perceptrons and convolutional neural networks. Here, our model predicts images and corresponding masks, and the masks are utilized for loss computation and network training to capture fine structural details and small components. In conjunction with model quantization and weight encoding, NeRVI yields highly compact compressive neural representations while preserving the image fidelity well. We demonstrate the effectiveness of NeRVI with isosurface rendering and direct volume rendering images generated from multiple data sets and compare NeRVI with other state-of-the-art deep learning-based (InSituNet, SIREN, NeRF, and NeRV) methods. Quantitative and qualitative results show that NeRVI provides an alternative solution that augments domain scientists' ability to manage, represent, and communicate scientific visualization output.

View Accepted Manuscript (DOE)

Research Organization:: University of Notre Dame, IN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); U.S. National Science Foundation

Grant/Contract Number:: SC0023145

OSTI ID:: 1995970

Alternate ID(s):: OSTI ID: 1997575

Journal Information:: Computers and Graphics, Journal Name: Computers and Graphics Vol. 116; ISSN 0097-8493

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (37)

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis Mildenhall, Ben; Srinivasan, Pratul P.; Tancik, Matthew Computer Vision – ECCV 2020 https://doi.org/10.1007/978-3-030-58452-8_24	book	November 2020
Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction Chabra, Rohan; Lenssen, Jan E.; Ilg, Eddy Computer Vision – ECCV 2020 https://doi.org/10.1007/978-3-030-58526-6_36	book	October 2020
Convolutional Occupancy Networks Peng, Songyou; Niemeyer, Michael; Mescheder, Lars Computer Vision – ECCV 2020 https://doi.org/10.1007/978-3-030-58580-8_31	book	January 2020
ESRGAN-based visualization for large-scale volume data Jiao, Chenyue; Bi, Chongke; Yang, Lu Journal of Visualization, Vol. 26, Issue 3 https://doi.org/10.1007/s12650-022-00891-2	journal	November 2022
Multilayer feedforward networks are universal approximators Hornik, Kurt; Stinchcombe, Maxwell; White, Halbert Neural Networks, Vol. 2, Issue 5 https://doi.org/10.1016/0893-6080(89)90020-8	journal	January 1989
Cinema image-based in situ analysis and visualization of MPAS-ocean simulations O’Leary, Patrick; Ahrens, James; Jourdain, Sébastien Parallel Computing, Vol. 55 https://doi.org/10.1016/j.parco.2015.10.005	journal	July 2016
Ionization Front Instabilities in Primordial H ii Regions Whalen, Daniel; Norman, Michael L. The Astrophysical Journal, Vol. 673, Issue 2 https://doi.org/10.1086/524400	journal	February 2008
Tracking and visualizing turbulent 3D features Silver, D. IEEE Transactions on Visualization and Computer Graphics, Vol. 3, Issue 2 https://doi.org/10.1109/2945.597796	journal	January 1997
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network Shi, Wenzhe; Caballero, Jose; Huszar, Ferenc 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.207	conference	June 2016
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric Zhang, Richard; Isola, Phillip; Efros, Alexei A. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00068	conference	June 2018
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference Jacob, Benoit; Kligys, Skirmantas; Chen, Bo 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2018.00286	conference	June 2018
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation Park, Jeong Joon; Florence, Peter; Straub, Julian 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00025	conference	June 2019
Occupancy Networks: Learning 3D Reconstruction in Function Space Mescheder, Lars; Oechsle, Michael; Niemeyer, Michael 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00459	conference	June 2019
Learning Implicit Fields for Generative Shape Modeling Chen, Zhiqin; Zhang, Hao 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00609	conference	June 2019
SAL: Sign Agnostic Learning of Shapes From Raw Data Atzmon, Matan; Lipman, Yaron 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00264	conference	June 2020
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision Niemeyer, Michael; Mescheder, Lars; Oechsle, Michael 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00356	conference	June 2020
Local Implicit Grid Representations for 3D Scenes Jiang, Chiyu; Sud, Avneesh; Makadia, Ameesh 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00604	conference	June 2020
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis Chan, Eric R.; Monteiro, Marco; Kellnhofer, Petr 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR46437.2021.00574	conference	June 2021
A Method for the Construction of Minimum-Redundancy Codes Huffman, David Proceedings of the IRE, Vol. 40, Issue 9 https://doi.org/10.1109/JRPROC.1952.273898	journal	September 1952
Space-time volumetric depth images for in-situ visualization Fernandes, Oliver; Frey, Steffen; Sadlo, Filip 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2014.7013205	conference	November 2014
Distributed Volumetric Neural Representation for in situ Visualization and Analysis Wu, Qi; Insley, Joseph A.; Mateevitsi, Victor A. 2022 IEEE 12th Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV57265.2022.9966405	conference	October 2022
Explorable images for visualizing volume data Tikhonova, Anna; Correa, Carlos D.; Ma, Kwan-Liu 2010 IEEE Pacific Visualization Symposium (PacificVis) https://doi.org/10.1109/PACIFICVIS.2010.5429595	conference	March 2010
An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis Ahrens, James; Jourdain, Sebastien; OLeary, Patrick SC14: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2014.40	conference	November 2014
Explorable Volumetric Depth Images from Raycasting Frey, Steffen; Sadlo, Filip; Ertl, Thomas 2013 XXVI SIBGRAPI - Conference on Graphics, Patterns and Images (SIBGRAPI), 2013 XXVI Conference on Graphics, Patterns and Images https://doi.org/10.1109/SIBGRAPI.2013.26	conference	August 2013
Image Quality Assessment: From Error Visibility to Structural Similarity Wang, Z.; Bovik, A. C.; Sheikh, H. R. IEEE Transactions on Image Processing, Vol. 13, Issue 4 https://doi.org/10.1109/TIP.2003.819861	journal	April 2004
Visualization by Proxy: A Novel Framework for Deferred Interaction with Volume Data Tikhonova, A.; Correa, C. D. IEEE Transactions on Visualization and Computer Graphics, Vol. 16, Issue 6 https://doi.org/10.1109/TVCG.2010.215	journal	November 2010
A Generative Model for Volume Rendering Berger, Matthew; Li, Jixian; Levine, Joshua A. IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 4 https://doi.org/10.1109/TVCG.2018.2816059	journal	April 2019
DL4SciVis: A State-of-the-Art Survey on Deep Learning for Scientific Visualization Wang, Chaoli; Han, Jun IEEE Transactions on Visualization and Computer Graphics, Vol. 29, Issue 8 https://doi.org/10.1109/TVCG.2022.3167896	journal	August 2023
CoordNet: Data Generation and Visualization Generation for Time-Varying Volumes via a Coordinate-Based Neural Network Han, Jun; Wang, Chaoli IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2022.3197203	journal	January 2022
State of the Art on Neural Rendering Tewari, A.; Fried, O.; Thies, J. Computer Graphics Forum, Vol. 39, Issue 2 https://doi.org/10.1111/cgf.14022	journal	May 2020
Compressive Neural Representations of Volumetric Scalar Fields Lu, Y.; Jiang, K.; Levine, J. A. Computer Graphics Forum, Vol. 40, Issue 3 https://doi.org/10.1111/cgf.14295	journal	June 2021
Advances in Neural Rendering Tewari, A.; Thies, J.; Mildenhall, B. Computer Graphics Forum, Vol. 41, Issue 2 https://doi.org/10.1111/cgf.14507	journal	May 2022
Neural Flow Map Reconstruction Sahoo, S.; Lu, Y.; Berger, M. Computer Graphics Forum, Vol. 41, Issue 3 https://doi.org/10.1111/cgf.14549	journal	June 2022
Fast Neural Representations for Direct Volume Rendering Weiss, S.; Hermüller, P.; Westermann, R. Computer Graphics Forum, Vol. 41, Issue 6 https://doi.org/10.1111/cgf.14578	journal	June 2022
An Exploratory Technique for Coherent Visualization of Time-varying Volume Data Tikhonova, A.; Correa, C. D.; Ma, K. -L. Computer Graphics Forum, Vol. 29, Issue 3 https://doi.org/10.1111/j.1467-8659.2009.01690.x	journal	June 2010
Experimental and Numerical Study of the Turbulence Characteristics of Airflow around a Research Vessel Popinet, Stéphane; Smith, Murray; Stevens, Craig Journal of Atmospheric and Oceanic Technology, Vol. 21, Issue 10 https://doi.org/10.1175/1520-0426(2004)021<1575:EANSOT>2.0.CO;2	journal	October 2004
Exploratory Lagrangian-Based Particle Tracing Using deep Learning Han, Mengjiao; Sane, Sudhanshu; Johnson, Chris R. Journal of Flow Visualization and Image Processing, Vol. 29, Issue 3 https://doi.org/10.1615/JFlowVisImageProc.2022041197	journal	January 2022

Similar Records

ShapeShop: Towards Understanding Deep Learning Representations via Interactive Experimentation

Conference · Tue May 30 00:00:00 EDT 2017 · OSTI ID:1361010

CoordNet: Data Generation and Visualization Generation for Time-Varying Volumes via a Coordinate-Based Neural Network

Journal Article · Mon Aug 08 00:00:00 EDT 2022 · IEEE Transactions on Visualization and Computer Graphics · OSTI ID:2345684

Detecting operons in bacterial genomes via visual representation learning

Journal Article · Thu Jan 21 23:00:00 EST 2021 · Scientific Reports · OSTI ID:1785368

Related Subjects

97 MATHEMATICS AND COMPUTING
Deep learning
Image compression
Implicit neural representation
Volume visualization

NeRVI: Compressive neural representation of visualization images for communicating volume visualization results

Citation Formats

References (37)

Similar Records

Related Subjects