Reducing Concurrency Bottlenecks in Parallel I/O Workloads
- Los Alamos National Laboratory
To enable high performance parallel checkpointing we introduced the Parallel Log Structured File System (PLFS). PLFS is middleware interposed on the file system stack to transform concurrent writing of one application file into many non-concurrently written component files. The promising effectiveness of PLFS makes it important to examine its performance for workloads other than checkpoint capture, notably the different ways that state snapshots may be later read, to make the case for using PLFS in the Exascale I/O stack. Reading a PLFS file involved reading each of its component files. In this paper we identify performance limitations on broader workloads in an early version of PLFS, specifically the need to build and distribute an index for the overall file, and the pressure on the underlying parallel file system's metadata server, and show how PLFS's decomposed components architecture can be exploited to alleviate bottlenecks in the underlying parallel file system.
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- DOE/LANL
- DOE Contract Number:
- AC52-06NA25396
- OSTI ID:
- 1012628
- Report Number(s):
- LA-UR-11-10426; TRN: US201110%%271
- Resource Relation:
- Conference: SC 2011 ; 2011-11-12 - 2011-11-18 ; Seattle, Washington, United States
- Country of Publication:
- United States
- Language:
- English
Similar Records
...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats
Characterizing output bottlenecks in a supercomputer