Online and Scalable Data Compression Pipeline with Guarantees on Quantities of Interest

Conference · Sun Oct 01 04:00:00 EDT 2023

DOI:https://doi.org/10.1109/e-Science58273.2023.10254934· OSTI ID:2251609

Banerjee, Tania ^[1]; Lee, Jaemoon ^[1]; Choi, Jong Youl ^[2]; Gong, Qian ^[2]; Chen, Jieyang ^[2]; Chang, C.S. ^[3]; Klasky, Scott ^[2]; Rangarajan, Anand ^[1]; Ranka, Sanjay ^[1]

---

1. University of Florida
2. ORNL
3. Princeton Plasma Physics Laboratory (PPPL)

Data compression is becoming critical for data-intensive scientific applications. Scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). Prior work has shown that a pipeline can be built to guarantee error on the primary data (PD) within user-defined bounds and achieve near-floating point QoI errors. In this paper, we present novel computational approaches for accelerating the pipeline and demonstrate results that enable concurrent execution of compression in parallel with the simulation nodes. This allows compression, including the writing of the required compression data, for the previous time step to be completed while the simulation proceeds with the current time step. Overall, the approach presented in this paper results in a 6–8 times improvement in computational overhead compared to previous work. These results were obtained using data generated by a large-scale fusion code called XGC, which produces hundreds of terabytes of data in a single day.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 2251609
• Country of Publication: United States
• Language: English