Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Conference · 01 November 2021

DOI:https://doi.org/10.1145/3458817.3476179· OSTI ID:1855631

Liang, Xin ^[1]; Gong, Qian ^[2]; Chen, Jieyang ^[2]; Whitney, Ben ^[2]; Wan, Lipeng ^[2]; Liu, Qing Gary ^[2]; Pugmire, Dave ^[2]; Archibald, Richard ^[2]; Podhorszki, Norbert ^[2]; Klasky, Scott ^[2]

---

1. Missouri University of Science and Technology
2. ORNL

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE; USDOE Office of Science (SC)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1855631
• Country of Publication: United States
• Language: English