A General Framework for Progressive Data Compression and Retrieval

Journal Article · 02 November 2023 · IEEE Transactions on Visualization and Computer Graphics

DOI: https://doi.org/10.1109/tvcg.2023.3327186 · OSTI ID:2204923

Magri, Victor Paludetto  ^[1]; Lindstrom, Peter  ^[1]

---

1. Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

In scientific simulations, observations, and experiments, the transfer of data to and from disk and across networks has become a major bottleneck for data analysis and visualization. Compression techniques have been employed to tackle this challenge, but traditional lossy methods often demand conservative error tolerances to meet the numerical accuracy requirements of both anticipated and unknown data analysis tasks. Progressive data compression and retrieval has emerged as a promising solution, where each analysis task dictates its own accuracy needs. However, few analysis algorithms inherently support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. Here, this paper presents a framework that enables progressive-precision data queries for any data compressor or numerical representation. Our strategy hinges on a multi-component representation that successively reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented this approach with four established scientific data compressors and assessed its effectiveness using real-world data sets from the SDRBench collection. The results show that our framework competes in accuracy with the standalone compressors it is based upon. Additionally, (de)compression time is proportional to the number of components requested by the user. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

• Research Organization: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• Grant/Contract Number: AC52-07NA27344
• OSTI ID: 2204923
• Report Number(s): LLNL--JRNL-852372; 1079555
• Journal Information: IEEE Transactions on Visualization and Computer Graphics, Journal Name: IEEE Transactions on Visualization and Computer Graphics Journal Issue: 1 Vol. 30; ISSN 1077-2626
• Publisher: IEEECopyright Statement
• Country of Publication: United States
• Language: English