Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

Conference · Wed May 01 00:00:00 EDT 2024

DOI:https://doi.org/10.23919/ISC.2024.10528927· OSTI ID:2439834

Venkatesh, Ranjan Sarpangala ^[1]; Eisenhauer, Greg ^[2]; Podhorszki, Norbert ^[3]; Ganyushin, Dmitry ^[3]; Klasky, Scott ^[3]; Gavrilovska, Ada ^[2]

---

1. Georgia Tech
2. Georgia Institute of Technology, Atlanta
3. ORNL

In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 2439834
• Country of Publication: United States
• Language: English