Use cases of lossy compression for floating-point data in scientific data sets

Cappello, Franck; Di, Sheng; Li, Sihuan; Liang, Xin; Gok, Ali Murat; Tao, Dingwen; Yoon, Chun Hong; Wu, Xin-Chuan; Alexeev, Yuri; Chong, Frederic T.

doi:10.1177/1094342019853336

Title: Use cases of lossy compression for floating-point data in scientific data sets

Journal Article · 09 July 2019 · International Journal of High Performance Computing Applications

DOI: https://doi.org/10.1177/1094342019853336 · OSTI ID:1560791

^[1]; Di, Sheng ^[2]; Li, Sihuan ^[3]; Liang, Xin ^[3]; Gok, Ali Murat ^[4]; Tao, Dingwen ^[5]; Yoon, Chun Hong ^[6]; Wu, Xin-Chuan ^[7]; Alexeev, Yuri ^[2]; Chong, Frederic T. ^[7]

Argonne National Lab. (ANL), Lemont, IL (United States); Univ. of Illinois, Urbana-Champaign, IL (United States)
Argonne National Lab. (ANL), Lemont, IL (United States)
Univ. of California, Riverside, CA (United States)
Northwestern Univ., Evanston, IL (United States)
Univ. of Alabama, Tuscaloosa, AL (United States)
SLAC National Accelerator Lab., Menlo Park, CA (United States)
Univ. of Chicago, IL (United States)

Architectural and technological trends of systems used for scientific computing call for a significant reduction of scientific data sets that are composed mainly of floating-point data. Here, this article surveys and presents experimental results of currently identified use cases of generic lossy compression to address the different limitations of scientific computing systems. The article shows from a collection of experiments run on parallel systems of a leadership facility that lossy data compression not only can reduce the footprint of scientific data sets on storage but also can reduce I/O and checkpoint/restart times, accelerate computation, and even allow significantly larger problems to be run than without lossy compression. In conclusion, these results suggest that lossy compression will become an important technology in many aspects of high performance scientific computing. Because the constraints for each use case are different and often conflicting, this collection of results also indicates the need for more specialization of the compression pipelines.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC); USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC02-76SF00515; AC02-06CH11357

OSTI ID:: 1560791

Journal Information:: International Journal of High Performance Computing Applications, Journal Name: International Journal of High Performance Computing Applications Journal Issue: 6 Vol. 33; ISSN 1094-3420

Publisher:: SAGECopyright Statement

Country of Publication:: United States

Language:: English

References (46)

General atomic and molecular electronic structure system Schmidt, Michael W.; Baldridge, Kim K.; Boatz, Jerry A. Journal of Computational Chemistry, Vol. 14, Issue 11, p. 1347-1363 https://doi.org/10.1002/jcc.540141112	journal	November 1993
Toward a Multi-method Approach: Lossy Data Compression for Climate Simulation Data Baker, Allison H.; Xu, Haiying; Hammerling, Dorit M. Lecture Notes in Computer Science https://doi.org/10.1007/978-3-319-67630-2_3	book	January 2017
Multilevel techniques for compression and reduction of scientific data—the univariate case Ainsworth, Mark; Tugluk, Ozan; Whitney, Ben Computing and Visualization in Science, Vol. 19, Issue 5-6 https://doi.org/10.1007/s00791-018-00303-9	journal	November 2018
Lossy compression of TPC data and trajectory tracking efficiency for the ALICE experiment Nicolaucig, A.; Ivanov, M.; Mattavelli, M. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 500, Issue 1-3 https://doi.org/10.1016/S0168-9002(03)00343-7	journal	March 2003
The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery Peters, S. M.; Kitaeff, V. V. Astronomy and Computing, Vol. 6 https://doi.org/10.1016/j.ascom.2014.06.003	journal	October 2014
Data compression in the petascale astronomy era: A GERLUMPH case study Vohl, D.; Fluke, C. J.; Vernardos, G. Astronomy and Computing, Vol. 12 https://doi.org/10.1016/j.ascom.2015.05.003	journal	September 2015
Efficient TPC data compression by track and cluster modeling Röhrich, Dieter; Vestbø, Anders Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 566, Issue 2 https://doi.org/10.1016/j.nima.2006.06.056	journal	October 2006
Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression Vohl, Dany; Pritchard, Tyler; Andreoni, Igor Publications of the Astronomical Society of Australia, Vol. 34 https://doi.org/10.1017/pasa.2017.34	journal	January 2017
Selenium single-wavelength anomalous diffraction de novo phasing using an X-ray-free electron laser Hunter, Mark S.; Yoon, Chun Hong; DeMirci, Hasan Nature Communications, Vol. 7, Issue 1 https://doi.org/10.1038/ncomms13388	journal	November 2016
The Paris Climate Agreement and future sea-level rise from Antarctica DeConto, Robert M.; Pollard, David; Alley, Richard B. Nature, Vol. 593, Issue 7857 https://doi.org/10.1038/s41586-021-03427-0	journal	May 2021
Se-SAD serial femtosecond crystallography datasets from selenobiotinyl-streptavidin Yoon, Chun Hong; DeMirci, Hasan; Sierra, Raymond G. Scientific Data, Vol. 4, Issue 1 https://doi.org/10.1038/sdata.2017.55	journal	April 2017
Compression of interferometric radio-astronomical data Offringa, A. R. Astronomy & Astrophysics, Vol. 595 https://doi.org/10.1051/0004-6361/201629565	journal	November 2016
The JPEG still picture compression standard Wallace, G. K. IEEE Transactions on Consumer Electronics, Vol. 38, Issue 1 https://doi.org/10.1109/30.125072	journal	January 1992
In-depth exploration of single-snapshot lossy compression techniques for N-body simulations Tao, Dingwen; Di, Sheng; Chen, Zizhong 2017 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2017.8257962	conference	December 2017
Optimizing Lossy Compression with Adjacent Snapshots for N-body Simulation Data Li, Sihuan; Di, Sheng; Liang, Xin 2018 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2018.8622101	conference	December 2018
Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets Liang, Xin; Di, Sheng; Tao, Dingwen 2018 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2018.8622520	conference	December 2018
Improving I/O Forwarding Throughput with Data Compression Welton, Benjamin; Kimpe, Dries; Cope, Jason 2011 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2011.80	conference	September 2011
PaSTRI: Error-Bounded Lossy Compression for Two-Electron Integrals in Quantum Chemistry Gok, Ali Murat; Di, Sheng; Alexeev, Yuri 2018 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2018.00013	conference	September 2018
Lossless compression of high-volume numerical data from simulations Engelson, V.; Fritzson, D.; Fritzson, P. Proceedings DCC 2000. Data Compression Conference https://doi.org/10.1109/DCC.2000.838221	conference	January 2000
Scheduling the I/O of HPC Applications Under Congestion Gainaru, Ana; Aupy, Guillaume; Benoit, Anne 2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2015.116	conference	May 2015
Exploration of Lossy Compression for Application-Level Checkpoint/Restart Sasaki, Naoto; Sato, Kento; Endo, Toshio 2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2015.67	conference	May 2015
Fast Error-Bounded Lossy HPC Data Compression with SZ Di, Sheng; Cappello, Franck 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.11	conference	May 2016
Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization Tao, Dingwen; Di, Sheng; Chen, Zizhong 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2017.115	conference	May 2017
The History of Storage Systems Goda, K.; Kitsuregawa, M. Proceedings of the IEEE, Vol. 100, Issue Special Centennial Issue https://doi.org/10.1109/JPROC.2012.2189787	journal	May 2012
Spatio-Temporal Just Noticeable Distortion Profile for Grey Scale Image/Video in DCT Domain IEEE Transactions on Circuits and Systems for Video Technology, Vol. 19, Issue 3 https://doi.org/10.1109/TCSVT.2009.2013518	journal	March 2009
Fast and Efficient Compression of Floating-Point Data Lindstrom, Peter; Isenburg, Martin IEEE Transactions on Visualization and Computer Graphics, Vol. 12, Issue 5 https://doi.org/10.1109/TVCG.2006.143	journal	September 2006
Fixed-Rate Compressed Floating-Point Arrays Lindstrom, Peter IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346458	journal	December 2014
MCREngine: A scalable checkpointing system using data-aware aggregation and compression Islam, Tanzima Zerin; Mohror, Kathryn; Bagchi, Saurabh 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/sc.2012.77	conference	November 2012
Data Reduction Techniques for Simulation, Visualization and Data Analysis: Survey on Scientific Data Reduction Techniques Li, S.; Marsaglia, N.; Garth, C. Computer Graphics Forum, Vol. 37, Issue 6 https://doi.org/10.1111/cgf.13336	journal	March 2018
The JPEG still picture compression standard Wallace, Gregory K. Communications of the ACM, Vol. 34, Issue 4 https://doi.org/10.1145/103085.103089	journal	April 1991
FTI: high performance fault tolerance interface for hybrid systems Bautista-Gomez, Leonardo; Tsuboi, Seiji; Komatitsch, Dimitri Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063427	conference	January 2011
HACC: extreme scaling and performance across diverse architectures Habib, Salman; Insley, Joe; Daniel, David Communications of the ACM, Vol. 60, Issue 1 https://doi.org/10.1145/3015569	journal	December 2016
18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios Fu, Haohuan; Yin, Wanwang; Yang, Guangwen Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17 https://doi.org/10.1145/3126908.3126910	conference	January 2017
Improving performance of iterative methods by lossy checkponting Tao, Dingwen; Di, Sheng; Liang, Xin Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '18 https://doi.org/10.1145/3208040.3208050	conference	January 2018
A first order approximation to the optimum checkpoint interval Young, John W. Communications of the ACM, Vol. 17, Issue 9 https://doi.org/10.1145/361147.361115	journal	September 1974
McrEngine: A Scalable Checkpointing System Using Data-Aware Aggregation and Compression Islam, Tanzima Zerin; Mohror, Kathryn; Bagchi, Saurabh Scientific Programming, Vol. 21, Issue 3-4 https://doi.org/10.1155/2013/341672	journal	January 2013
The Community Earth System Model: A Framework for Collaborative Research Hurrell, James W.; Holland, M. M.; Gent, P. R. Bulletin of the American Meteorological Society, Vol. 94, Issue 9 https://doi.org/10.1175/BAMS-D-12-00121.1	journal	September 2013
Exploring the feasibility of lossy compression for PDE simulations Calhoun, Jon; Cappello, Franck; Olson, Luke N. The International Journal of High Performance Computing Applications, Vol. 33, Issue 2 https://doi.org/10.1177/1094342018762036	journal	November 2017
Data systems for the Linac coherent light source Thayer, J.; Damiani, D.; Ford, C. Advanced Structural and Chemical Imaging, Vol. 3, Issue 1 https://doi.org/10.1186/s40679-016-0037-7	journal	January 2017
Wavefield compression for adjoint methods in full-waveform inversion Boehm, Christian; Hanzich, Mauricio; de la Puente, Josep GEOPHYSICS, Vol. 81, Issue 6 https://doi.org/10.1190/geo2015-0653.1	journal	November 2016
GZIP file format specification version 4.3 Deutsch, P. https://doi.org/10.17487/rfc1952	report	May 1996
Detailed Modeling, Design, and Evaluation of a Scalable Multi-level Checkpointing System Moody, A.; Bronevetsky, G.; Mohror, K. https://doi.org/10.2172/984082	report	April 2010
The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery Peters, Sean M.; Kitaeff, Vyacheslav V. arXiv https://doi.org/10.48550/arxiv.1401.7433	preprint	January 2014
Data Compression in the Petascale Astronomy Era: a GERLUMPH case study Vohl, Dany; Fluke, Christopher J.; Vernardos, Georgios arXiv https://doi.org/10.48550/arxiv.1505.05617	text	January 2015
Improving Performance of Iterative Methods by Lossy Checkponting Tao, Dingwen; Di, Sheng; Liang, Xin arXiv https://doi.org/10.48550/arxiv.1804.11268	text	January 2018
Evaluating lossy data compression on climate simulation data within a large ensemble Baker, Allison H.; Hammerling, Dorit M.; Mickelson, Sheri A. Geoscientific Model Development, Vol. 9, Issue 12 https://doi.org/10.5194/gmd-9-4381-2016	journal	January 2016

Cited By (2)

Significantly improving lossy compression quality based on an optimized hybrid prediction model Liang, Xin; Di, Sheng; Li, Sihuan SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356193	conference	November 2019
Compression Challenges in Large Scale Partial Differential Equation Solvers Götschel, Sebastian; Weiser, Martin Algorithms, Vol. 12, Issue 9 https://doi.org/10.3390/a12090197	journal	September 2019

Similar Records

Exploring the feasibility of lossy compression for PDE simulations

Journal Article · 2018 · International Journal of High Performance Computing Applications · OSTI ID:1425688

Calhoun, Jon; Cappello, Franck; Olson, Luke N.; +2 more

FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework for Scientific Floating-point Data

Conference · 2019 · OSTI ID:1804070

Underwood, Robert; Di, Sheng; Calhoun, Jon C.; +1 more

Lossy checkpoint compression in full waveform inversion: a case study with ZFPv0.5.5 and the overthrust model

Journal Article · 2022 · Geoscientific Model Development (Online) · OSTI ID:1889198

Kukreja, Navjot; Hückelheim, Jan; Louboutin, Mathias; +3 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Lossy compression
applications
floating-point data
scientific data set
use cases

Title: Use cases of lossy compression for floating-point data in scientific data sets

Citation Formats

References (46)

Cited By (2)

Similar Records

Related Subjects