Collective input/output under memory constraints
Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.
- Research Organization:
- Argonne National Laboratory (ANL), Argonne, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC); National Science Foundation (NSF)
- DOE Contract Number:
- AC02-06CH11357
- OSTI ID:
- 1392900
- Journal Information:
- International Journal of High Performance Computing Applications, Vol. 29, Issue 1; ISSN 1094-3420
- Publisher:
- SAGE
- Country of Publication:
- United States
- Language:
- English
The International Exascale Software Project roadmap
|
journal | January 2011 |
Energy-Efficient Computing for Extreme-Scale Science
|
journal | November 2009 |
Design and Evaluation of Multiple-Level Data Staging for Blue Gene Systems
|
journal | June 2011 |
Bridging the Gap Between Parallel File Systems and Local File Systems: A Case Study with PVFS
|
conference | September 2008 |
Disk-directed I/O for MIMD multiprocessors
|
journal | February 1997 |
Similar Records
Opportunities for nonvolatile memory systems in extreme-scale high-performance computing
Proactive Data Containers for Scientific Storage (Final Report)