skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Data Reduction Techniques for Simulation, Visualization and Data Analysis

Abstract

Data reduction is increasingly being applied to scientific data for numerical simulations, scientific visualizations, and data analyses. It is most often used to lower I/O and storage costs, and sometimes to lower in-memory data size as well. With this work, we consider five categories of data reduction techniques based on their information loss: 1) truly lossless, 2) near lossless, 3) lossy, 4) mesh reduction, and 5) derived representations. We then survey available techniques in each of these categories, summarize their properties from a practical point of view, and discuss relative merits within a category. We believe, in total, this work will enable simulation scientists and visualization/data analysis scientists to decide which data reduction techniques will be most helpful for their needs.

Authors:
ORCiD logo [1];  [2];  [3];  [4];  [5];  [2]
  1. National Center for Atmospheric Research, Boulder, CO (United States); Univ. of Oregon, Eugene, OR (United States)
  2. Univ. of Oregon, Eugene, OR (United States)
  3. Univ. of Kaiserslautern (Germany)
  4. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  5. National Center for Atmospheric Research, Boulder, CO (United States)
Publication Date:
Research Org.:
Univ. of Oregon, Eugene, OR (United States); Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1463451
Grant/Contract Number:  
SC0010652
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Computer Graphics Forum
Additional Journal Information:
Journal Volume: 37; Journal Issue: 6; Journal ID: ISSN 0167-7055
Publisher:
Wiley
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; data reduction techniques; simulation; data analysis; survey

Citation Formats

Li, Shaomeng, Marsaglia, Nicole, Garth, Christoph, Woodring, Jonathan, Clyne, John, and Childs, Hank. Data Reduction Techniques for Simulation, Visualization and Data Analysis. United States: N. p., 2018. Web. doi:10.1111/cgf.13336.
Li, Shaomeng, Marsaglia, Nicole, Garth, Christoph, Woodring, Jonathan, Clyne, John, & Childs, Hank. Data Reduction Techniques for Simulation, Visualization and Data Analysis. United States. https://doi.org/10.1111/cgf.13336
Li, Shaomeng, Marsaglia, Nicole, Garth, Christoph, Woodring, Jonathan, Clyne, John, and Childs, Hank. 2018. "Data Reduction Techniques for Simulation, Visualization and Data Analysis". United States. https://doi.org/10.1111/cgf.13336. https://www.osti.gov/servlets/purl/1463451.
@article{osti_1463451,
title = {Data Reduction Techniques for Simulation, Visualization and Data Analysis},
author = {Li, Shaomeng and Marsaglia, Nicole and Garth, Christoph and Woodring, Jonathan and Clyne, John and Childs, Hank},
abstractNote = {Data reduction is increasingly being applied to scientific data for numerical simulations, scientific visualizations, and data analyses. It is most often used to lower I/O and storage costs, and sometimes to lower in-memory data size as well. With this work, we consider five categories of data reduction techniques based on their information loss: 1) truly lossless, 2) near lossless, 3) lossy, 4) mesh reduction, and 5) derived representations. We then survey available techniques in each of these categories, summarize their properties from a practical point of view, and discuss relative merits within a category. We believe, in total, this work will enable simulation scientists and visualization/data analysis scientists to decide which data reduction techniques will be most helpful for their needs.},
doi = {10.1111/cgf.13336},
url = {https://www.osti.gov/biblio/1463451}, journal = {Computer Graphics Forum},
issn = {0167-7055},
number = 6,
volume = 37,
place = {United States},
year = {2018},
month = {3}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 3 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Streaming Simplification of Tetrahedral Meshes
journal, January 2007


In-situ Sampling of a Large-Scale Particle Simulation for Interactive Visualization and Analysis
journal, June 2011


Multidimensional Directional Filter Banks and Surfacelets
journal, April 2007


Least squares quantization in PCM
journal, March 1982


Adaptive Multilinear Tensor Product Wavelets
journal, January 2016


Volume rendering of DCT-based compressed 3D scalar data
journal, March 1995


Transform Coding for Hardware-accelerated Volume Rendering
journal, November 2007


Differential FCM: increasing value prediction accuracy by improving table usage efficiency
conference, January 2001

  • Goeman, B.; Vandierendonck, H.; de Bosschere, K.
  • HPCA-7 - 7th IEEE Symposium on High Performance Computer Architecture, Proceedings HPCA Seventh International Symposium on High-Performance Computer Architecture
  • https://doi.org/10.1109/HPCA.2001.903264

Embedded image coding using zerotrees of wavelet coefficients
journal, January 1993


Simplex and Diamond Hierarchies: Models and Applications
journal, November 2011


Feature-Based Statistical Analysis of Combustion Simulation Data
journal, December 2011


A Survey of Topology-based Methods in Visualization
journal, June 2016


In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms
journal, June 2016


Biorthogonal bases of compactly supported wavelets
journal, June 1992


Efficient, Low-Complexity Image Coding With a Set-Partitioning Embedded Block Coder
journal, November 2004


Rapid High Quality Compression of Volume Data for Visualization
journal, September 2001


Wavelet Transforms That Map Integers to Integers
journal, July 1998


Wavelet-Based 3D Compression Scheme for Interactive Visualization of Very Large Volume Data
journal, March 1999


Fast Discrete Curvelet Transforms
journal, January 2006


An Information-Aware Framework for Exploring Multivariate Data Sets
journal, December 2013


Arithmetic coding for data compression
journal, June 1987


Generalized unstructured decimation [computer graphics]
journal, January 1996


Fast and Efficient Compression of Floating-Point Data
journal, September 2006


ISABELA for effective in situ compression of scientific data: ISABELA FOR EFFECTIVE
journal, July 2012

  • Lakshminarasimhan, Sriram; Shah, Neil; Ethier, Stephane
  • Concurrency and Computation: Practice and Experience, Vol. 25, Issue 4
  • https://doi.org/10.1002/cpe.2887

Three-dimensional subband coding of video using the zero-tree method
conference, February 1996


Compression of individual sequences via variable-rate coding
journal, September 1978


Lossless compression of volume data
conference, January 1994


An Algorithm for Vector Quantizer Design
journal, January 1980


A universal algorithm for sequential data compression
journal, May 1977


Vector quantization for volume rendering
conference, January 1992


BTRFS: The Linux B-Tree Filesystem
journal, August 2013


Seismic data compression using high-dimensional wavelet transforms
conference, January 1996


A mathematical theory of communication
journal, January 2001


A Method for the Construction of Minimum-Redundancy Codes
journal, September 1952


Simplification of tetrahedral meshes with error bounds
journal, January 1999


Parallel Tensor Compression for Large-Scale Scientific Data
conference, May 2016


A prototype discovery environment for analyzing and visualizing terascale turbulent fluid flow simulations
conference, March 2005


Ueber die stetige Abbildung einer Line auf ein Fl�chenst�ck
journal, September 1891


The predictability of data values
conference, January 1997


Interactive Exploration and Analysis of Large-Scale Simulations Using Topology-Based Data Segmentation
journal, September 2011


Lossy volume compression using Tucker truncation and thresholding
journal, May 2015


Quadric-based simplification in any dimension
journal, April 2005


Reducing disk storage of full-3D seismic waveform tomography (F3DT) through lossy online compression
journal, August 2016


Factoring wavelet transforms into lifting steps
journal, May 1998


R-trees: a dynamic index structure for spatial searching
conference, January 1984


Visualization by Proxy: A Novel Framework for Deferred Interaction with Volume Data
journal, November 2010


Adaptive tetrapuzzles: efficient out-of-core construction and visualization of gigantic multiresolution polygonal models
journal, August 2004


Compressed progressive meshes
journal, January 2000


FPC: A High-Speed Compressor for Double-Precision Floating-Point Data
journal, January 2009


The R*-tree: an efficient and robust access method for points and rectangles
journal, May 1990


Image-driven simplification
journal, July 2000


ISOBAR Preconditioner for Effective and High-throughput Lossless Data Compression
conference, April 2012

  • Schendel, Eric R.; Jin, Ye; Shah, Neil
  • 2012 IEEE International Conference on Data Engineering (ICDE 2012), 2012 IEEE 28th International Conference on Data Engineering
  • https://doi.org/10.1109/ICDE.2012.114

Three-Dimensional Embedded Subband Coding with Optimized Truncation (3-D ESCOT)
journal, May 2001


Advanced techniques for high-quality multi-resolution volume rendering
journal, February 2004


Partitioning a Large Simulation as It Runs
journal, July 2016


Four-dimensional wavelet compression of arbitrarily sized echocardiographic data
journal, September 2002


A Combined Eulerian-Lagrangian Data Representation for Large-Scale Applications
journal, October 2017


Explorable Volumetric Depth Images from Raycasting
conference, August 2013

  • Frey, Steffen; Sadlo, Filip; Ertl, Thomas
  • 2013 XXVI SIBGRAPI - Conference on Graphics, Patterns and Images (SIBGRAPI), 2013 XXVI Conference on Graphics, Patterns and Images
  • https://doi.org/10.1109/SIBGRAPI.2013.26

Efficient query processing on unstructured tetrahedral meshes
conference, January 2006

  • Papadomanolakis, Stratos; Ailamaki, Anastassia; Lopez, Julio C.
  • Proceedings of the 2006 ACM SIGMOD international conference on Management of data - SIGMOD '06
  • https://doi.org/10.1145/1142473.1142535

Survey and analysis of multiresolution methods for turbulence data
journal, February 2016


Organization and maintenance of large ordered indexes
journal, January 1972


An Adaptive Prediction-Based Approach to Lossless Compression of Floating-Point Volume Data
journal, December 2012


Discrete Cosine Transform
journal, January 1974


Fixed-Rate Compressed Floating-Point Arrays
journal, December 2014


Tensor Decompositions and Applications
journal, August 2009


Enabling Adaptive Scientific Workflows Via Trigger Detection
conference, January 2015

  • Salloum, Maher; Bennett, Janine C.; Pinar, Ali
  • Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV2015
  • https://doi.org/10.1145/2828612.2828619

Interactive, Internet Delivery of Visualization via Structured Prerendered Multiresolution Imagery
journal, March 2008


A parallel multiresolution volume rendering algorithm for large data visualization
journal, February 2005


Query-Driven Visualization of Time-Varying Adaptive Mesh Refinement Data
journal, November 2008


Decimation of triangle meshes
journal, July 1992


Frequency domain volume rendering by the wavelet X-ray transform
journal, July 2000


Lossless compression of predicted floating-point geometry
journal, July 2005


Real-Time Synthesis of Compression Algorithms for Scientific Data
conference, November 2016

  • Burtscher, Martin; Mukka, Hari; Yang, Annie
  • SC16: International Conference for High Performance Computing, Networking, Storage and Analysis
  • https://doi.org/10.1109/SC.2016.22

Bitmap index design and evaluation
journal, June 1998


High performance scalable image compression with EBCOT
journal, July 2000


An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis
conference, November 2014

  • Ahrens, James; Jourdain, Sebastien; OLeary, Patrick
  • SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
  • https://doi.org/10.1109/SC.2014.40

Fast volume rendering of compressed data
conference, January 1993


The Visible Human Project
journal, March 1998


Out-of-core compression and decompression of large n-dimensional scalar fields
journal, September 2003


An Application of Multivariate Statistical Analysis for Query-Driven Visualization
journal, March 2011


Transparent in Situ Data Transformations in ADIOS
conference, May 2014


QccPack: an open-source software library for quantization, compression, and coding
conference, December 2000


Direct rendering of Laplacian pyramid compressed volume data
conference, January 1995


Evaluating the efficacy of wavelet configurations on turbulent-flow data
conference, October 2015


Using feature importance metrics to detect events of interest in scientific computing applications
conference, October 2017


Spatiotemporal Wavelet Compression for Visualization of Scientific Simulation Data
conference, September 2017


Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization
conference, May 2017


Salient time steps selection from large scale time-varying data sets with dynamic time warping
conference, October 2012


Wavelets applied to lossless compression and progressive transmission of floating point data in 3-D curvilinear grids
conference, January 1996


Revisiting wavelet compression for large-scale climate data using JPEG 2000 and ensuring data precision
conference, October 2011


MPC: A Massively Parallel Compression Algorithm for Scientific Data
conference, September 2015


A Mathematical Theory of Communication
journal, July 1948


Adaptive tetrapuzzles: efficient out-of-core construction and visualization of gigantic multiresolution polygonal models
conference, January 2004


The R*-tree: an efficient and robust access method for points and rectangles
conference, January 1990

  • Beckmann, Norbert; Kriegel, Hans-Peter; Schneider, Ralf
  • Proceedings of the 1990 ACM SIGMOD international conference on Management of data - SIGMOD '90
  • https://doi.org/10.1145/93597.98741

Decimation of triangle meshes
conference, January 1992

  • Schroeder, William J.; Zarge, Jonathan A.; Lorensen, William E.
  • Proceedings of the 19th annual conference on Computer graphics and interactive techniques - SIGGRAPH '92
  • https://doi.org/10.1145/133994.134010

Bitmap index design and evaluation
conference, January 1998


A Mathematical Theory of Communication
journal, October 1948


A method for the construction of minimum-redundancy codes
journal, February 2006


The Visible Human Project: a resource for education
journal, January 1999


High performance scalable image compression with EBCOT
conference, January 1999


Adaptive TetraPuzzles: efficient out-of-core construction and visualization of gigantic multiresolution polygonal models
conference, January 2008


Lossy volume compression using Tucker truncation and thresholding
text, January 2016


Works referencing / citing this record:

Use cases of lossy compression for floating-point data in scientific data sets
journal, May 2019


Is Smaller Always Better? - Evaluating Video Compression Techniques for Simulation Ensembles
text, January 2021


Visitation Graphs: Interactive Ensemble Visualization with Visitation Maps
text, January 2021


VAPOR: A Visualization Package Tailored to Analyze Simulation Data in Earth System Science
journal, August 2019