Organizing Large Data Sets for Efficient Analyses on HPC Systems

Conference · Fri Apr 01 04:00:00 EDT 2022

DOI:https://doi.org/10.1088/1742-6596/2224/1/012042· OSTI ID:1871082

Gu, Junmin ^[1]; Davis, Philip ^[2]; Eisenhauer, Greg ^[3]; Godoy, William ^[4]; Huebl, Axel ^[1]; Klasky, Scott ^[4]; Parashar, Manish ^[2]; Podhorszki, Norbert ^[4]; Poeschel, Franz ^[5]; Vay, Jean-Luc ^[1]; Wan, Lipeng ^[4]; Wang, Ruonan ^[4]; Wu, Kesheng ^[1]

---

1. Lawrence Berkeley National Laboratory (LBNL)
2. University of Utah
3. Georgia Institute of Technology, Atlanta
4. ORNL
5. Helmholtz-Zentrum Dresden Rossendorf (HDZR), Germany

Upcoming exascale applications could introduce significant data management challenges due to their large sizes, dynamic work distribution, and involvement of accelerators such as graphical processing units, GPUs. In this work, we explore the performance of reading and writing operations involving one such scientific application on two different supercomputers. Our tests showed that the Adaptable Input and Output System, ADIOS, was able to achieve speeds over 1TB/s, a significant fraction of the peak I/O performance on Summit. We also demonstrated the querying functionality in ADIOS could effectively support common selective data analysis operations, such as conditional histograms. In tests, this query mechanism was able to reduce the execution time by a factor of five. More importantly, ADIOS data management framework allows us to achieve these performance improvements with only a minimal amount of coding effort.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE; USDOE Office of Science (SC)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1871082
• Country of Publication: United States
• Language: English