Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints

Journal Article · Thu Jul 28 00:00:00 EDT 2022 · IEEE Transactions on Parallel and Distributed Systems

DOI:https://doi.org/10.1109/tpds.2022.3194695· OSTI ID:1908133

Liu, Yuanjian ^[1]; Di, Sheng ^[2]; Zhao, Kai ^[3]; Jin, Sian ^[4]; Wang, Cheng ^[2]; Chard, Kyle ^[1]; Tao, Dingwen ^[4]; Foster, Ian ^[1]; Cappello, Franck ^[2]

---

1. Univ. of Chicago, IL (United States)
2. Argonne National Lab. (ANL), Lemont, IL (United States)
3. Univ. of California, Riverside, CA (United States)
4. Washington State Univ., Pullman, WA (United States)

Vast volumes of data are produced by today's scientific simulations and advanced instruments. These data cannot be stored and transferred efficiently because of limited I/O bandwidth, network speed, and storage capacity. Error-bounded lossy compression can be an effective method for addressing these issues: not only can it significantly reduce data size, but it can also control the data distortion based on user-defined error bounds. In practice, many scientific applications have specific requirements or constraints for lossy compression, in order to guarantee that the reconstructed data are valid for post hoc analysis. For example, some datasets contain irrelevant data that should be isolated in particular and users often have intuition regarding value ranges, geospatial regions, and other data subsets that are crucial for subsequent analysis. Existing state-of-the-art error-bounded lossy compressors, however, do not consider these constraints during compression, resulting in inferior compression ratios with respect to user's post hoc analysis, due to the fact that the data itself provides little or no value for post hoc analysis. In this work we address this issue by proposing an optimized framework that can preserve diverse constraints during the error-bounded lossy compression, e.g., cleaning the irrelevant data, efficiently preserving different precision for multiple value intervals, and allowing users to set diverse precision over both regular and irregular regions. We perform our evaluation on a supercomputer with up to 2,100 cores. Experiments with six real-world applications show that our proposed diverse constraints based error-bounded lossy compressor can obtain a higher visual quality or data fidelity on reconstructed data with the same or even higher compression ratios compared with the traditional state-of-the-art compressor SZ. Furthermore, our experiments also demonstrate very good scalability in compression performance compared with the I/O throughput of the parallel file system.

• Research Organization: Argonne National Laboratory (ANL), Argonne, IL (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); National Science Foundation (NSF); Exascale Computing Project (ECP)
• Grant/Contract Number: AC02-06CH11357
• OSTI ID: 1908133
• Journal Information: IEEE Transactions on Parallel and Distributed Systems, Journal Name: IEEE Transactions on Parallel and Distributed Systems Journal Issue: 12 Vol. 33; ISSN 1045-9219
• Publisher: IEEE
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Big Data
Data Reduction
Error-Bounded Lossy Compression
Large-Scale Scientific Simulation