Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Use cases of lossy compression for floating-point data in scientific data sets

Journal Article · · International Journal of High Performance Computing Applications
 [1];  [2];  [3];  [3];  [4];  [5];  [6];  [7];  [2];  [7]
  1. Argonne National Lab. (ANL), Lemont, IL (United States); Univ. of Illinois, Urbana-Champaign, IL (United States)
  2. Argonne National Lab. (ANL), Lemont, IL (United States)
  3. Univ. of California, Riverside, CA (United States)
  4. Northwestern Univ., Evanston, IL (United States)
  5. Univ. of Alabama, Tuscaloosa, AL (United States)
  6. SLAC National Accelerator Lab., Menlo Park, CA (United States)
  7. Univ. of Chicago, IL (United States)
+ Show Author Affiliations
Architectural and technological trends of systems used for scientific computing call for a significant reduction of scientific data sets that are composed mainly of floating-point data. Here, this article surveys and presents experimental results of currently identified use cases of generic lossy compression to address the different limitations of scientific computing systems. The article shows from a collection of experiments run on parallel systems of a leadership facility that lossy data compression not only can reduce the footprint of scientific data sets on storage but also can reduce I/O and checkpoint/restart times, accelerate computation, and even allow significantly larger problems to be run than without lossy compression. In conclusion, these results suggest that lossy compression will become an important technology in many aspects of high performance scientific computing. Because the constraints for each use case are different and often conflicting, this collection of results also indicates the need for more specialization of the compression pipelines.
Research Organization:
Argonne National Lab. (ANL), Argonne, IL (United States); SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)
Sponsoring Organization:
National Science Foundation (NSF); USDOE Exascale Computing Project; USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Grant/Contract Number:
AC02-06CH11357; AC02-76SF00515
OSTI ID:
1560791
Alternate ID(s):
OSTI ID: 1575218
Journal Information:
International Journal of High Performance Computing Applications, Journal Name: International Journal of High Performance Computing Applications Journal Issue: 6 Vol. 33; ISSN 1094-3420
Publisher:
SAGECopyright Statement
Country of Publication:
United States
Language:
English

References (46)

General atomic and molecular electronic structure system journal November 1993
Toward a Multi-method Approach: Lossy Data Compression for Climate Simulation Data book January 2017
Multilevel techniques for compression and reduction of scientific data—the univariate case journal November 2018
Lossy compression of TPC data and trajectory tracking efficiency for the ALICE experiment
  • Nicolaucig, A.; Ivanov, M.; Mattavelli, M.
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 500, Issue 1-3 https://doi.org/10.1016/S0168-9002(03)00343-7
journal March 2003
The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery journal October 2014
Data compression in the petascale astronomy era: A GERLUMPH case study journal September 2015
Efficient TPC data compression by track and cluster modeling
  • Röhrich, Dieter; Vestbø, Anders
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 566, Issue 2 https://doi.org/10.1016/j.nima.2006.06.056
journal October 2006
Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression journal January 2017
Selenium single-wavelength anomalous diffraction de novo phasing using an X-ray-free electron laser journal November 2016
The Paris Climate Agreement and future sea-level rise from Antarctica journal May 2021
Se-SAD serial femtosecond crystallography datasets from selenobiotinyl-streptavidin journal April 2017
Compression of interferometric radio-astronomical data journal November 2016
The JPEG still picture compression standard journal January 1992
In-depth exploration of single-snapshot lossy compression techniques for N-body simulations conference December 2017
Optimizing Lossy Compression with Adjacent Snapshots for N-body Simulation Data conference December 2018
Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets conference December 2018
Improving I/O Forwarding Throughput with Data Compression conference September 2011
PaSTRI: Error-Bounded Lossy Compression for Two-Electron Integrals in Quantum Chemistry conference September 2018
Lossless compression of high-volume numerical data from simulations conference January 2000
Scheduling the I/O of HPC Applications Under Congestion conference May 2015
Exploration of Lossy Compression for Application-Level Checkpoint/Restart conference May 2015
Fast Error-Bounded Lossy HPC Data Compression with SZ conference May 2016
Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization conference May 2017
The History of Storage Systems journal May 2012
Spatio-Temporal Just Noticeable Distortion Profile for Grey Scale Image/Video in DCT Domain journal March 2009
Fast and Efficient Compression of Floating-Point Data journal September 2006
Fixed-Rate Compressed Floating-Point Arrays journal December 2014
MCREngine: A scalable checkpointing system using data-aware aggregation and compression
  • Islam, Tanzima Zerin; Mohror, Kathryn; Bagchi, Saurabh
  • 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/sc.2012.77
conference November 2012
Data Reduction Techniques for Simulation, Visualization and Data Analysis: Survey on Scientific Data Reduction Techniques journal March 2018
The JPEG still picture compression standard journal April 1991
FTI: high performance fault tolerance interface for hybrid systems
  • Bautista-Gomez, Leonardo; Tsuboi, Seiji; Komatitsch, Dimitri
  • Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063427
conference January 2011
HACC: extreme scaling and performance across diverse architectures journal December 2016
18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios
  • Fu, Haohuan; Yin, Wanwang; Yang, Guangwen
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17 https://doi.org/10.1145/3126908.3126910
conference January 2017
Improving performance of iterative methods by lossy checkponting conference January 2018
A first order approximation to the optimum checkpoint interval journal September 1974
McrEngine: A Scalable Checkpointing System Using Data-Aware Aggregation and Compression journal January 2013
The Community Earth System Model: A Framework for Collaborative Research journal September 2013
Exploring the feasibility of lossy compression for PDE simulations journal November 2017
Data systems for the Linac coherent light source journal January 2017
Wavefield compression for adjoint methods in full-waveform inversion journal November 2016
GZIP file format specification version 4.3 report May 1996
Detailed Modeling, Design, and Evaluation of a Scalable Multi-level Checkpointing System report April 2010
The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery preprint January 2014
Data Compression in the Petascale Astronomy Era: a GERLUMPH case study text January 2015
Improving Performance of Iterative Methods by Lossy Checkponting text January 2018
Evaluating lossy data compression on climate simulation data within a large ensemble journal January 2016

Cited By (2)

Significantly improving lossy compression quality based on an optimized hybrid prediction model
  • Liang, Xin; Di, Sheng; Li, Sihuan
  • SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356193
conference November 2019
Compression Challenges in Large Scale Partial Differential Equation Solvers journal September 2019

Figures / Tables (18)

Table 1(p. 2)table Table 1
Figure 1(p. 3)figure Figure 1
Figure 2(p. 4)figure Figure 2
Figure 3(p. 5)figure Figure 3
Figure 4(p. 6)figure Figure 4
Figure 5(p. 7)figure Figure 5
Figure 6(p. 7)figure Figure 6
Table 2(p. 8)table Table 2
Table 3(p. 9)table Table 3
Figure 7(p. 10)figure Figure 7
Figure 8(p. 11)figure Figure 8
Figure 9(p. 12)figure Figure 9
Algorithm 1(p. 13)figure Algorithm 1
Table 4(p. 13)table Table 4
Figure 10(p. 14)figure Figure 10
Figure 11(p. 14)figure Figure 11
Figure 12(p. 15)figure Figure 12
Table 5(p. 16)table Table 5

Similar Records

Exploring the feasibility of lossy compression for PDE simulations
Journal Article · 2018 · International Journal of High Performance Computing Applications · OSTI ID:1425688

Related Subjects

97 MATHEMATICS AND COMPUTING
Lossy compression
applications
floating-point data
scientific data set
use cases