Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound
Abstract
An effective data compressor is becoming increasingly critical to today's scientific research, and many lossy compressors are developed in the context of absolute error bounds. Based on physical/chemical definitions of simulation fields or multiresolution demand, however, many scientific applications need to compress the data with a pointwise relative error bound (i.e., the smaller the data value, the smaller the compression error to tolerate). To this end, we propose two optimized lossy compression strategies under a state-of-the-art three-staged compression framework (prediction + quantization + entropy-encoding). The first strategy (called block-based strategy) splits the data set into many small blocks and computes an absolute error bound for each block, so it is particularly suitable for the data with relatively high consecutiveness in space. The second strategy (called multi-threshold-based strategy) splits the whole value range into multiple groups with exponentially increasing thresholds and performs the compression in each group separately, which is particularly suitable for the data with a relatively large value range and spiky value changes. We implement the two strategies rigorously and evaluate them comprehensively by using two scientific applications which both require lossy compression with point-wise relative error bound. Experiments show that the two strategies exhibit the best compression qualitiesmore »
- Authors:
-
[1];
[2];
- Argonne National Lab. (ANL), Lemont, IL (United States)
- Univ. of Alabama, Tuscaloosa, AL (United States)
- Univ. of California, Riverside, CA (United States)
- Publication Date:
- Research Org.:
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Org.:
- USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC); National Science Foundation (NSF)
- OSTI Identifier:
- 1510064
- Grant/Contract Number:
- AC02-06CH11357
- Resource Type:
- Accepted Manuscript
- Journal Name:
- IEEE Transactions on Parallel and Distributed Systems
- Additional Journal Information:
- Journal Volume: 30; Journal Issue: 2; Journal ID: ISSN 1045-9219
- Publisher:
- IEEE
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; Lossy compression; high performance computing; relative error bound; science data
Citation Formats
Di, Sheng, Tao, Dingwen, Liang, Xin, and Cappello, Franck. Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound. United States: N. p., 2018.
Web. doi:10.1109/TPDS.2018.2859932.
Di, Sheng, Tao, Dingwen, Liang, Xin, & Cappello, Franck. Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound. United States. https://doi.org/10.1109/TPDS.2018.2859932
Di, Sheng, Tao, Dingwen, Liang, Xin, and Cappello, Franck. Fri .
"Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound". United States. https://doi.org/10.1109/TPDS.2018.2859932. https://www.osti.gov/servlets/purl/1510064.
@article{osti_1510064,
title = {Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound},
author = {Di, Sheng and Tao, Dingwen and Liang, Xin and Cappello, Franck},
abstractNote = {An effective data compressor is becoming increasingly critical to today's scientific research, and many lossy compressors are developed in the context of absolute error bounds. Based on physical/chemical definitions of simulation fields or multiresolution demand, however, many scientific applications need to compress the data with a pointwise relative error bound (i.e., the smaller the data value, the smaller the compression error to tolerate). To this end, we propose two optimized lossy compression strategies under a state-of-the-art three-staged compression framework (prediction + quantization + entropy-encoding). The first strategy (called block-based strategy) splits the data set into many small blocks and computes an absolute error bound for each block, so it is particularly suitable for the data with relatively high consecutiveness in space. The second strategy (called multi-threshold-based strategy) splits the whole value range into multiple groups with exponentially increasing thresholds and performs the compression in each group separately, which is particularly suitable for the data with a relatively large value range and spiky value changes. We implement the two strategies rigorously and evaluate them comprehensively by using two scientific applications which both require lossy compression with point-wise relative error bound. Experiments show that the two strategies exhibit the best compression qualities on different types of data sets respectively. In conclusion, the compression ratio of our lossy compressor is higher than that of other state-of-the-art compressors by 17.2-618 percent on the climate simulation data and 30-210 percent on the N-body simulation data, with the same relative error bound and without degradation of the overall visualization effect of the entire data.},
doi = {10.1109/TPDS.2018.2859932},
journal = {IEEE Transactions on Parallel and Distributed Systems},
number = 2,
volume = 30,
place = {United States},
year = {Fri Jul 27 00:00:00 EDT 2018},
month = {Fri Jul 27 00:00:00 EDT 2018}
}
Search WorldCat to find libraries that may hold this journalWeb of Science
Works referencing / citing this record:
Significantly improving lossy compression quality based on an optimized hybrid prediction model
conference, November 2019
- Liang, Xin; Di, Sheng; Li, Sihuan
- SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis