Fast Algorithms for Scientific Data Compression

Conference · Fri Dec 01 04:00:00 EST 2023

DOI:https://doi.org/10.1109/HiPC58850.2023.00030· OSTI ID:2438668

Banerjee, Tania ^[1]; Lee, Jaemoon ^[1]; Choi, Jong Youl ^[2]; Gong, Qian ^[2]; Chen, Jieyang ^[3]; Klasky, Scott ^[2]; Rangarajan, Anand ^[1]; Ranka, Sanjay ^[1]

---

1. University of Florida
2. ORNL
3. University of Alabama, Birmingham

Many scientific simulations and experiments generate terabytes to petabytes of data daily, necessitating data compression techniques. Unlike video and image compression, scientists require methods that accurately preserve primary data (PD) and derived quantities of interest (QoIs). In our previous work, we demonstrated the effectiveness of hybrid compression techniques that combine machine learning with traditional approaches. This paper presents innovative computational techniques aimed at expediting the compression pipeline. Our experiments, conducted on two distinct platforms with a large-scale XGC-based fusion simulation, demonstrate that the overhead incurred by these new approaches is less than one percent of the computational resources needed for the simulation.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 2438668
• Country of Publication: United States
• Language: English