Leveraging Pre-Built Catalogs and Object-Level Scheduling to Eliminate I/O Bottlenecks in HPC Environments

Journal Article · Thu Mar 20 00:00:00 EDT 2025 · IEEE Access

DOI:https://doi.org/10.1109/ACCESS.2025.3553311· OSTI ID:2587474

Lee, Seoyeong ^[1]; Park, Junghwan ^[1]; Kim, Yoochan ^[1]; Jamil, Safdar ^[1]; Khan, Awais ^[2]; Woo Son, Seung ^[3]; Lee, Jae-Kook ^[4]; An, Do-Sik ^[4]; Hong, Taeyoung ^[4]; Kim, Youngjae ^[1]

---

1. Sogang University, Seoul (South Korea)
2. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
3. University of Massachusetts Lowell, Amherst, MA (United States)
4. Korea Institute of Science and Technology Information (KISTI), Daejeon (South Korea)

Modern High-Performance Computing (HPC) environments face mounting challenges due to the shift from large to small file datasets, along with an increasing number of users and parallelized applications. As HPC systems rely on Parallel File Systems (PFS), such as Lustre for data processing, performance bottlenecks stemming from Object Storage Target (OST) contention have become a significant concern. Existing solutions, such as LADS with its object-level scheduling approach, fall short in large-scale HPC environments due to their inability to effectively address metadata I/O bottlenecks and the growing number of I/O processes. This study highlights the pressing need for a comprehensive solution that tackles both OST contention and metadata I/O challenges in diverse HPC workloads. To address these challenges, we propose SwiftLoad, an object-level I/O scheduling framework that leverages a metadata catalog to enhance the performance and efficiency of parallel HPC utilities. The adoption of the metadata catalog mitigates the metadata I/O bottlenecks that commonly occur in HPC utilities, a challenge that is particularly pronounced in object-level I/O scheduling. SwiftLoad addresses OST contention and the uneven distribution of I/O processes across different OSTs through mathematical modeling and incorporates a Loader Configuration Module to regulate the number of I/O processes. Evaluated with two representative utilities—data deduplication profiling and data augmentation—SwiftLoad achieved performance improvements of up to 5.63x and 11.0x, respectively, on a production supercomputer.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC); Korean Government [Ministry of Science and ICT (MSIT)]; Korea Institute of Science and Technology Information
• Grant/Contract Number: AC05-00OR22725
• OSTI ID: 2587474
• Journal Information: IEEE Access, Journal Name: IEEE Access Vol. 13; ISSN 2169-3536
• Publisher: Institute of Electrical and Electronics Engineers (IEEE)
• Country of Publication: United States
• Language: English

Related Subjects

HPC
I/O
parallel file system
parallel processing