Battle of the Defaults: Extracting Performance Characteristics of HDF5 under Production Load
Popular parallel I/O libraries, such as HDF5, provide tuning parameters to obtain superior performance. However, the selection of effective parameters on production systems is complex due to the interdependence of I/O software and file system layers. Hence, application developers typically use the default parameters and often experience poor I/O performance. This work conducts a benchmarking-based analysis on the HDF5 behaviors with a wide variety of I/O patterns to extract performance characteristics under the production workload. To make the analysis well controlled, we exercise I/O benchmarks on POSIX-IO, MPI-IO, and HDF5 using the same I/O patterns and in the same jobs. To address high performance variability in production environments, we repeat the benchmarks across I/O patterns, storage devices, and time intervals. Based on the results, we identified consistent HDF5 behaviors that appropriate configurations and operations on dataset layout and file-metadata placement can improve performance significantly. We apply our findings and evaluate the tuned I/O library on two supercomputers: Summit and Cori. The results show that our tuned parameters can achieve more than 10× I/O performance speedup than that with default parameters on both systems, suggesting the effectiveness, stability, and generality of our solution.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1814317
- Resource Relation:
- Conference: The 21th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID) - Melbourne, , Australia - 5/10/2021 8:00:00 AM-5/13/2021 8:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
Scalable I/O Tracing and Analysis
...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats