Z-checker: A framework for assessing lossy compression of scientific data

Tao, Dingwen; Di, Sheng; Guo, Hanqi; Chen, Zizhong; Cappello, Franck

doi:10.1177/1094342017737147

Title: Z-checker: A framework for assessing lossy compression of scientific data

Journal Article · 15 November 2017 · International Journal of High Performance Computing Applications

DOI: https://doi.org/10.1177/1094342017737147 · OSTI ID:1437773

Tao, Dingwen ^[1]; Di, Sheng ^[2]; Guo, Hanqi ^[2]; Chen, Zizhong ^[3]; Cappello, Franck ^[4]

Univ. of California, Riverside, CA (United States)
Argonne National Lab. (ANL), Lemont, IL (United States)
Univ. of California, Riverside, CA (United States); Beijing Univ. of Technology, Beijing (China)
Argonne National Lab. (ANL), Lemont, IL (United States); Univ. of Illinois Urbana-Champaign, Champaign, IL (United States)

Because of the vast volume of data being produced by today's scientific simulations and experiments, lossy data compressor allowing user-controlled loss of accuracy during the compression is a relevant solution for significantly reducing the data size. However, lossy compressor developers and users are missing a tool to explore the features of scientific data sets and understand the data alteration after compression in a systematic and reliable way. To address this gap, we have designed and implemented a generic framework called Z-checker. On the one hand, Z-checker combines a battery of data analysis components for data compression. On the other hand, Z-checker is implemented as an open-source community tool to which users and developers can contribute and add new analysis components based on their additional analysis demands. In this study, we present a survey of existing lossy compressors. Then, we describe the design framework of Z-checker, in which we integrated evaluation metrics proposed in prior work as well as other analysis tools. Specifically, for lossy compressor developers, Z-checker can be used to characterize critical properties (such as entropy, distribution, power spectrum, principal component analysis, and autocorrelation) of any data set to improve compression strategies. For lossy compression users, Z-checker can detect the compression quality (compression ratio and bit rate) and provide various global distortion analysis comparing the original data with the decompressed data (peak signal-to-noise ratio, normalized mean squared error, rate-distortion, rate-compression error, spectral, distribution, and derivatives) and statistical analysis of the compression error (maximum, minimum, and average error; autocorrelation; and distribution of errors). Z-checker can perform the analysis with either coarse granularity (throughout the whole data set) or fine granularity (by user-defined blocks), such that the users and developers can select the best fit, adaptive compressors for different parts of the data set. Z-checker features a visualization interface displaying all analysis results in addition to some basic views of the data sets such as time series. To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific data sets.

View Journal Article

Cite

Export

Save

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 1437773

Alternate ID(s):: OSTI ID: 1510019

Journal Information:: International Journal of High Performance Computing Applications, Vol. 33, Issue 2; ISSN 1094-3420

Publisher:: SAGECopyright Statement

Country of Publication:: United States

Language:: English

References (19)

Exploration of Lossy Compression for Application-Level Checkpoint/Restart Sasaki, Naoto; Sato, Kento; Endo, Toshio 2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2015.67	conference	May 2015
Fast Error-Bounded Lossy HPC Data Compression with SZ Di, Sheng; Cappello, Franck 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.11	conference	May 2016
Fast Lossless Compression of Scientific Floating-Point Data Ratanaworabhan, P.; Ke, J.; Burtscher, M. Data Compression Conference (DCC'06) https://doi.org/10.1109/DCC.2006.35	conference	January 2006
Fixed-Rate Compressed Floating-Point Arrays Lindstrom, Peter IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346458	journal	December 2014
HACC: extreme scaling and performance across diverse architectures Habib, Salman; Insley, Joe; Daniel, David Communications of the ACM, Vol. 60, Issue 1 https://doi.org/10.1145/3015569	journal	December 2016
A study of the characteristics of white noise using the empirical mode decomposition method Wu, Zhaohua; Huang, Norden E. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, Vol. 460, Issue 2046 https://doi.org/10.1098/rspa.2003.1221	journal	June 2004
ISABELA for effective in situ compression of scientific data: ISABELA FOR EFFECTIVE Lakshminarasimhan, Sriram; Shah, Neil; Ethier, Stephane Concurrency and Computation: Practice and Experience, Vol. 25, Issue 4 https://doi.org/10.1002/cpe.2887	journal	July 2012
Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization Tao, Dingwen; Di, Sheng; Chen, Zizhong 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2017.115	conference	May 2017
The JPEG still picture compression standard Wallace, G. K. IEEE Transactions on Consumer Electronics, Vol. 38, Issue 1 https://doi.org/10.1109/30.125072	journal	January 1992
Assessing the Effects of Data Compression in Simulations Using Physically Motivated Metrics Laney, Daniel; Langer, Steven; Weber, Christopher Scientific Programming, Vol. 22, Issue 2 https://doi.org/10.1155/2014/835419	journal	January 2014
A methodology for evaluating the impact of data compression on climate simulation data Baker, Allison H.; Xu, Haiying; Dennis, John M. Proceedings of the 23rd international symposium on High-performance parallel and distributed computing - HPDC '14 https://doi.org/10.1145/2600212.2600217	conference	January 2014
The Earth System Grid: Supporting the Next Generation of Climate Modeling Research Bernholdt, D.; Bharathi, S.; Brown, D. Proceedings of the IEEE, Vol. 93, Issue 3 https://doi.org/10.1109/JPROC.2004.842745	journal	March 2005
A universal algorithm for sequential data compression Ziv, J.; Lempel, A. IEEE Transactions on Information Theory, Vol. 23, Issue 3 https://doi.org/10.1109/TIT.1977.1055714	journal	May 1977
NUMARCK: Machine Learning Algorithm for Resiliency and Checkpointing Chen, Zhengzhang; Son, Seung Woo; Hendrix, William SC14: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2014.65	conference	November 2014
Advanced Photon Source Austin, Elizabeth Synchrotron Radiation News, Vol. 29, Issue 2 https://doi.org/10.1080/08940886.2016.1149431	journal	March 2016
Industrial-era global ocean heat uptake doubles in recent decades Gleckler, Peter J.; Durack, Paul J.; Stouffer, Ronald J. Nature Climate Change, Vol. 6, Issue 4 https://doi.org/10.1038/nclimate2915	journal	January 2016
A Method for the Construction of Minimum-Redundancy Codes Huffman, David Proceedings of the IRE, Vol. 40, Issue 9 https://doi.org/10.1109/JRPROC.1952.273898	journal	September 1952
Industrial-era global ocean heat uptake doubles in recent decades Gleckler, Peter J.; Durack, Paul J.; Stouffer, Ronald J. Nature Climate Change, Vol. 6, Issue 4 https://doi.org/10.1038/nclimate2915	journal	January 2016
Evaluating lossy data compression on climate simulation data within a large ensemble Baker, Allison H.; Hammerling, Dorit M.; Mickelson, Sheri A. Geoscientific Model Development, Vol. 9, Issue 12 https://doi.org/10.5194/gmd-9-4381-2016	journal	January 2016