Tuning HDF5 for Lustre File Systems
Conference
·
OSTI ID:1050648
HDF5 is a cross-platform parallel I/O library that is used by a wide variety of HPC applications for the flexibility of its hierarchical object-database representation of scientific data. We describe our recent work to optimize the performance of the HDF5 and MPI-IO libraries for the Lustre parallel file system. We selected three different HPC applications to represent the diverse range of I/O requirements, and measured their performance on three different systems to demonstrate the robustness of our optimizations across different file system configurations and to validate our optimization strategy. We demonstrate that the combined optimizations improve HDF5 parallel I/O performance by up to 33 times in some cases running close to the achievable peak performance of the underlying file system and demonstrate scalable performance up to 40,960-way concurrency.
- Research Organization:
- Ernest Orlando Lawrence Berkeley National Laboratory, Berkeley, CA (US)
- Sponsoring Organization:
- Computational Research Division
- DOE Contract Number:
- AC02-05CH11231
- OSTI ID:
- 1050648
- Report Number(s):
- LBNL-4803E
- Country of Publication:
- United States
- Language:
- English
Similar Records
Tuning HDF5 subfiling performance on parallel file systems
Exploiting Lustre File Joining for Effective Collective IO
GPU Direct I/O with HDF5
Conference
·
Fri May 12 00:00:00 EDT 2017
·
OSTI ID:1398484
Exploiting Lustre File Joining for Effective Collective IO
Conference
·
Sun Dec 31 23:00:00 EST 2006
·
OSTI ID:931507
GPU Direct I/O with HDF5
Conference
·
Sun Nov 01 00:00:00 EDT 2020
·
OSTI ID:1760260