Silo & HDF5 I/O Scaling Improvements on BG/P Systems
Silo and HDF5 are I/O libraries used by many codes important to the LLNL's Weapons and Complex Integration (WCI) mission. In the past year, modest adjustments and tuning of Silo, HDF5 and the I/O configuration of the BG/P platform, Dawn, were undertaken. A key goal of this work was to improve I/O performance without requiring any changes in the application codes themselves. In particular, the application codes have been allowed to continue to use a simplified yet highly flexible I/O paradigm known as 'Poor Man's Parallel I/O', where scalability is achieved through concurrent, serial I/O to multiple files. The results demonstrate substantial performance gains (better than 50x in many cases) at large scale (greater than 64,000 MPI tasks). They describe key enhancements made to Silo, HDF5 and the I/O configuration of our BG/P platform and present very favorable results from scalability studies over a wide range of operating scenarios.
- Research Organization:
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- W-7405-ENG-48
- OSTI ID:
- 1018434
- Report Number(s):
- LLNL-CONF-463573; TRN: US201114%%1
- Resource Relation:
- Conference: Presented at: NECDC 2010, Los Alamos, NM, United States, Oct 18 - Oct 22, 2010
- Country of Publication:
- United States
- Language:
- English
Similar Records
Tuning HDF5 for Lustre File Systems
PSIP For HDF5 Pilot Project (Final Report)