Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)
Abstract
Abstract. Geoscientific models and measurements generate false precision (scientifically meaningless data bits) that wastes storage space. False precision can mislead (by implying noise is signal) and be scientifically pointless, especially for measurements. By contrast, lossy compression can be both economical (save space) and heuristic (clarify data limitations) without compromising the scientific integrity of data. Data quantization can thus be appropriate regardless of whether space limitations are a concern. We introduce, implement, and characterize a new lossy compression scheme suitable for IEEE floating-point data. Our new Bit Grooming algorithm alternately shaves (to zero) and sets (to one) the least significant bits of consecutive values to preserve a desired precision. This is a symmetric, two-sided variant of an algorithm sometimes called Bit Shaving that quantizes values solely by zeroing bits. Our variation eliminates the artificial low bias produced by always zeroing bits, and makes Bit Grooming more suitable for arrays and multi-dimensional fields whose mean statistics are important. Bit Grooming relies on standard lossless compression to achieve the actual reduction in storage space, so we tested Bit Grooming by applying the DEFLATE compression algorithm to bit-groomed and full-precision climate data stored in netCDF3, netCDF4, HDF4, and HDF5 formats. Bit Grooming reduces themore »
- Authors:
- Publication Date:
- Research Org.:
- Univ. of California, Irvine (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC)
- OSTI Identifier:
- 1328486
- Alternate Identifier(s):
- OSTI ID: 1363945
- Grant/Contract Number:
- SC0012998
- Resource Type:
- Published Article
- Journal Name:
- Geoscientific Model Development (Online)
- Additional Journal Information:
- Journal Name: Geoscientific Model Development (Online) Journal Volume: 9 Journal Issue: 9; Journal ID: ISSN 1991-9603
- Publisher:
- Copernicus Publications, EGU
- Country of Publication:
- Germany
- Language:
- English
- Subject:
- 58 GEOSCIENCES
Citation Formats
Zender, Charles S. Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+). Germany: N. p., 2016.
Web. doi:10.5194/gmd-9-3199-2016.
Zender, Charles S. Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+). Germany. https://doi.org/10.5194/gmd-9-3199-2016
Zender, Charles S. Mon .
"Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)". Germany. https://doi.org/10.5194/gmd-9-3199-2016.
@article{osti_1328486,
title = {Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)},
author = {Zender, Charles S.},
abstractNote = {Abstract. Geoscientific models and measurements generate false precision (scientifically meaningless data bits) that wastes storage space. False precision can mislead (by implying noise is signal) and be scientifically pointless, especially for measurements. By contrast, lossy compression can be both economical (save space) and heuristic (clarify data limitations) without compromising the scientific integrity of data. Data quantization can thus be appropriate regardless of whether space limitations are a concern. We introduce, implement, and characterize a new lossy compression scheme suitable for IEEE floating-point data. Our new Bit Grooming algorithm alternately shaves (to zero) and sets (to one) the least significant bits of consecutive values to preserve a desired precision. This is a symmetric, two-sided variant of an algorithm sometimes called Bit Shaving that quantizes values solely by zeroing bits. Our variation eliminates the artificial low bias produced by always zeroing bits, and makes Bit Grooming more suitable for arrays and multi-dimensional fields whose mean statistics are important. Bit Grooming relies on standard lossless compression to achieve the actual reduction in storage space, so we tested Bit Grooming by applying the DEFLATE compression algorithm to bit-groomed and full-precision climate data stored in netCDF3, netCDF4, HDF4, and HDF5 formats. Bit Grooming reduces the storage space required by initially uncompressed and compressed climate data by 25–80 and 5–65 %, respectively, for single-precision values (the most common case for climate data) quantized to retain 1–5 decimal digits of precision. The potential reduction is greater for double-precision datasets. When used aggressively (i.e., preserving only 1–2 digits), Bit Grooming produces storage reductions comparable to other quantization techniques such as Linear Packing. Unlike Linear Packing, whose guaranteed precision rapidly degrades within the relatively narrow dynamic range of values that it can compress, Bit Grooming guarantees the specified precision throughout the full floating-point range. Data quantization by Bit Grooming is irreversible (i.e., lossy) yet transparent, meaning that no extra processing is required by data users/readers. Hence Bit Grooming can easily reduce data storage volume without sacrificing scientific precision or imposing extra burdens on users.},
doi = {10.5194/gmd-9-3199-2016},
journal = {Geoscientific Model Development (Online)},
number = 9,
volume = 9,
place = {Germany},
year = {Mon Sep 19 00:00:00 EDT 2016},
month = {Mon Sep 19 00:00:00 EDT 2016}
}
https://doi.org/10.5194/gmd-9-3199-2016
Web of Science
Works referenced in this record:
Validation of SO 2 retrievals from the Ozone Monitoring Instrument over NE China
journal, January 2008
- Krotkov, Nickolay A.; McClure, Brittany; Dickerson, Russell R.
- Journal of Geophysical Research, Vol. 113, Issue D16
Analysis of self-describing gridded geoscience data with netCDF Operators (NCO)
journal, October 2008
- Zender, Charles S.
- Environmental Modelling & Software, Vol. 23, Issue 10-11
Mineral Dust Entrainment and Deposition (DEAD) model: Description and 1990s dust climatology
journal, January 2003
- Zender, Charles S.
- Journal of Geophysical Research, Vol. 108, Issue D14
A High Performance Compression Method for Climate Data
conference, August 2014
- Liu, Songbin; Huang, Xiaomeng; Ni, Yufang
- 2014 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA)
FPC: A High-Speed Compressor for Double-Precision Floating-Point Data
journal, January 2009
- Burtscher, Martin; Ratanaworabhan, Paruj
- IEEE Transactions on Computers, Vol. 58, Issue 1
Lossless compression of predicted floating-point geometry
journal, July 2005
- Isenburg, Martin; Lindstrom, Peter; Snoeyink, Jack
- Computer-Aided Design, Vol. 37, Issue 8
Compression of individual sequences via variable-rate coding
journal, September 1978
- Ziv, J.; Lempel, A.
- IEEE Transactions on Information Theory, Vol. 24, Issue 5
CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model
journal, November 2011
- Dennis, John M.; Edwards, Jim; Evans, Katherine J.
- The International Journal of High Performance Computing Applications, Vol. 26, Issue 1
Scaling Properties of Common Statistical Operators for Gridded Datasets
journal, November 2007
- Zender, Charles S.; Mangalam, Harry
- The International Journal of High Performance Computing Applications, Vol. 21, Issue 4
A universal algorithm for sequential data compression
journal, May 1977
- Ziv, J.; Lempel, A.
- IEEE Transactions on Information Theory, Vol. 23, Issue 3
Netcdf Operators (Nco) Version 4.6.1
code, September 2016
- Zender, Charles S.
- Zenodo, 1 ZIP file (4.9 MB)
Finding the Goldilocks zone: Compression-error trade-off for large gridded datasets
journal, July 2016
- Silver, Jeremy D.; Zender, Charles S.
- Geoscientific Model Development Discussions
MERRA: NASA’s Modern-Era Retrospective Analysis for Research and Applications
journal, July 2011
- Rienecker, Michele M.; Suarez, Max J.; Gelaro, Ronald
- Journal of Climate, Vol. 24, Issue 14