The compression–error trade-off for large gridded data sets

Silver, Jeremy D.; Zender, Charles S.

doi:10.5194/gmd-10-413-2017

The compression–error trade-off for large gridded data sets

Journal Article · Thu Jan 26 23:00:00 EST 2017 · Geoscientific Model Development (Online)

DOI:https://doi.org/10.5194/gmd-10-413-2017· OSTI ID:1341345

;

The netCDF-4 format is widely used for large gridded scientific data sets and includes several compression methods: lossy linear scaling and the non-lossy deflate and shuffle algorithms. Many multidimensional geoscientific data sets exhibit considerable variation over one or several spatial dimensions (e.g., vertically) with less variation in the remaining dimensions (e.g., horizontally). On such data sets, linear scaling with a single pair of scale and offset parameters often entails considerable loss of precision. We introduce an alternative compression method called "layer-packing" that simultaneously exploits lossy linear scaling and lossless compression. Layer-packing stores arrays (instead of a scalar pair) of scale and offset parameters. An implementation of this method is compared with lossless compression, storing data at fixed relative precision (bit-grooming) and scalar linear packing in terms of compression ratio, accuracy and speed.

When viewed as a trade-off between compression and error, layer-packing yields similar results to bit-grooming (storing between 3 and 4 significant figures). Bit-grooming and layer-packing offer significantly better control of precision than scalar linear packing. Relative performance, in terms of compression and errors, of bit-groomed and layer-packed data were strongly predicted by the entropy of the exponent array, and lossless compression was well predicted by entropy of the original data array. Layer-packed data files must be "unpacked" to be readily usable. The compression and precision characteristics make layer-packing a competitive archive format for many scientific data sets.

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)

Grant/Contract Number:: SC0012998

OSTI ID:: 1341345

Journal Information:: Geoscientific Model Development (Online), Journal Name: Geoscientific Model Development (Online) Journal Issue: 1 Vol. 10; ISSN 1991-9603

Publisher:: Copernicus Publications, EGUCopyright Statement

Country of Publication:: Germany

Language:: English

References (12)

Rate-Distortion Theory Berger, Toby Wiley Encyclopedia of Telecommunications https://doi.org/10.1002/0471219282.eot142	book	January 2003
The ERA-Interim reanalysis: configuration and performance of the data assimilation system Dee, D. P.; Uppala, S. M.; Simmons, A. J. Quarterly Journal of the Royal Meteorological Society, Vol. 137, Issue 656 https://doi.org/10.1002/qj.828	journal	April 2011
Analysis of self-describing gridded geoscience data with netCDF Operators (NCO) Zender, Charles S. Environmental Modelling & Software, Vol. 23, Issue 10-11 https://doi.org/10.1016/j.envsoft.2008.03.004	journal	October 2008
The JPEG 2000 still image compression standard Skodras, A.; Christopoulos, C.; Ebrahimi, T. IEEE Signal Processing Magazine, Vol. 18, Issue 5 https://doi.org/10.1109/79.952804	journal	January 2001
Fast Error-Bounded Lossy HPC Data Compression with SZ Di, Sheng; Cappello, Franck 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.11	conference	May 2016
A Method for the Construction of Minimum-Redundancy Codes Huffman, David Proceedings of the IRE, Vol. 40, Issue 9 https://doi.org/10.1109/JRPROC.1952.273898	journal	September 1952
Revisiting wavelet compression for large-scale climate data using JPEG 2000 and ensuring data precision Woodring, Jonathan; Mniszewski, Susan; Brislawn, Christopher 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092314	conference	October 2011
A universal algorithm for sequential data compression Ziv, J.; Lempel, A. IEEE Transactions on Information Theory, Vol. 23, Issue 3 https://doi.org/10.1109/TIT.1977.1055714	journal	May 1977
Compression of individual sequences via variable-rate coding Ziv, J.; Lempel, A. IEEE Transactions on Information Theory, Vol. 24, Issue 5 https://doi.org/10.1109/TIT.1978.1055934	journal	September 1978
What every computer scientist should know about floating-point arithmetic Goldberg, David ACM Computing Surveys, Vol. 23, Issue 1 https://doi.org/10.1145/103162.103163	journal	March 1991
A methodology for evaluating the impact of data compression on climate simulation data Baker, Allison H.; Xu, Haiying; Dennis, John M. Proceedings of the 23rd international symposium on High-performance parallel and distributed computing - HPDC '14 https://doi.org/10.1145/2600212.2600217	conference	January 2014
Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+) Zender, Charles S. Geoscientific Model Development, Vol. 9, Issue 9 https://doi.org/10.5194/gmd-9-3199-2016	journal	January 2016

Similar Records

Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)

Journal Article · Mon Sep 19 00:00:00 EDT 2016 · Geoscientific Model Development (Online) · OSTI ID:1328486

The compression–error trade-off for large gridded data sets

Citation Formats

References (12)

Similar Records

Related Subjects