IOrchestrator: improving the performance of multi-node I/O systems via inter-server coordination
- Los Alamos National Laboratory
- WAYNE STATE UNIV
- WAYNE STATE UNIV.
A cluster of I/O nodes and a parallel file system are often used to provide high-throughput I/O service to a parallel compute cluster. To exploit I/O parallelism parallel file systems stripe file data across the I/O nodes. While this practice is effective in serving asynchronous requests, it may break individual program's spatial locality, which can seriously degrade I/O performance when the I/O nodes concurrently serve synchronous requests from multiple I/O-intensive programs. In this paper we propose a scheme, Orchestrator, to improve the I/O performance of multi-I/O-node systems by orchestrating I/O services among programs when such inter-I/O-node coordination is dynamically determined to be cost effective. We have implemented IOrchestrator in the PVFS2 parallel file system. Our experiments with representative parallel benchmarks show that IOrchestrator can significantly improve I/O performance - by up to a factor of 2.5 - delivered by a cluster of I/O nodes servicing concurrently-running parallel programs. Notably, we have not observed any scenarios in which the use of IOrchestrator causes significant performance degradation.
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC52-06NA25396
- OSTI ID:
- 1009541
- Report Number(s):
- LA-UR-10-02601; LA-UR-10-2601; TRN: US201106%%973
- Resource Relation:
- Conference: Supercomputing 2010 ; November 13, 2010 ; New Orleans, LA
- Country of Publication:
- United States
- Language:
- English
Similar Records
HPC Global File System Performance Analysis Using A Scientific-Application Derived Benchmark
PVFS : a parallel file system for linux clusters