Reproducibility and Variability of I/O Performance on BG/Q: Lessons Learned from a Data Aggregation Algorithm
- Argonne National Lab. (ANL), Argonne, IL (United States)
Reading and writing data efficiently from different tiers of storage is necessary for most scientific simulations to achieve good performance at scale. Many software solutions have been developed to decrease the I/O bottleneck. One wellknown strategy, in the context of collective I/O operations, is the two-phase I/O scheme. This strategy consists of selecting a subset of processes to aggregate contiguous pieces of data before performing reads/writes. In our previous work, we implemented the two-phase I/O scheme with a MPI-based topology-aware algorithm. Our algorithm showed very good performance at scale compared to the standard I/O libraries such as POSIX I/O and MPI I/O. However, the algorithm had several limitations hindering a satisfying reproducibility of our experiments. In this paper, we extend our work by 1) identifying the obstacles we face to reproduce our experiments and 2) discovering solutions that reduce the unpredictability of our results.
- Research Organization:
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science - Office of Advanced Scientific Computing Research
- DOE Contract Number:
- AC02-06CH11357
- OSTI ID:
- 1414287
- Report Number(s):
- ANL-/ALCF-17/9; 140351
- Country of Publication:
- United States
- Language:
- English
Similar Records
VisIO: enabling interactive visualization of ultra-scale, time-series data via high-bandwidth distributed I/O systems
Performance Comparison of GPFS 1.3 and GPFS 1.4 for POSIX and MPI-IO