Skel: Generative Software for Producing Skeletal I/O Applications
Massively parallel computations consist of a mixture of computation, communication, and I/O. As part of the co-design for the inevitable progress towards exascale computing, we must apply lessons learned from past work to succeed in this new age of computing. Of the three components listed above, implementing an effective parallel I/O solution has often been overlooked by application scientists and was usually added to large scale simulations only when existing serial techniques had failed. As scientists teams scaled their codes to run on hundreds of processors, it was common to call on an I/O expert to implement a set of more scalable I/O routines. These routines were easily separated from the calculations and communication, and in many cases, an I/O kernel was derived from the application which could be used for testing I/O performance independent of the application. These I/O kernels developed a life of their own used as a broad measure for comparing different I/O techniques. Unfortunately, as years passed and computation and communication changes required changes to the I/O, the separate I/O kernel used for benchmarking remained static no longer providing an accurate indicator of the I/O performance of the simulation making I/O research less relevant for the application scientists. In this paper we describe a new approach to this problem where I/O kernels are replaced with skeletal I/O applications automatically generated from an abstract set of simulation I/O parameters. We realize this abstraction by leveraging the ADIOS middleware's XML I/O specification with additional runtime parameters. Skeletal applications offer all of the benefits of I/O kernels including allowing I/O optimizations to focus on useful I/O patterns. Moreover, since they are automatically generated, it is easy to produce an updated I/O skeleton whenever the simulation's I/O changes. In this paper we analyze the performance of automatically generated I/O skeletal applications for the S3D and GTS codes. We show that these skeletal applications achieve performance comparable to that of the production applications. We wrap up the paper with a discussion of future changes to make the skeletal application better approximate the actual I/O performed in the simulation.
- Research Organization:
- National Renewable Energy Laboratory (NREL), Golden, CO (United States)
- Sponsoring Organization:
- USDOE Office of Science, Advanced Scientific Computing Research; National Science Foundation; USDOE Grant, SciDAC Fusion Simulation Prototype Center for Plasma Edge Simulation; USDOE NNSA
- DOE Contract Number:
- AC36-08GO28308
- OSTI ID:
- 1039097
- Report Number(s):
- NREL/CP-2C00-54732
- Journal Information:
- Proceedings of the 2011 IEEE Seventh International Conference on e-Science Workshops (eScienceW), 5-8 December 2011, Stockholm, Sweden, Journal Name: Proceedings of the 2011 IEEE Seventh International Conference on e-Science Workshops (eScienceW), 5-8 December 2011, Stockholm, Sweden
- Publisher:
- Piscataway, NJ: Institute of Electrical and Electronics Engineers (IEEE)
- Country of Publication:
- United States
- Language:
- English
Similar Records
Performance Impact of I/O on QMCPack Simulations at the Petascale and Beyond
Data Management Challenges of Exascale Scientific Simulations: A Case Study with the Gyrokinetic Toroidal Code and ADIOS
Related Subjects
COMMUNICATIONS
COMPUTER CODES
I/O routines
KERNELS
MIXTURES
PARALLEL PROCESSING
PERFORMANCE
PRODUCTION
SIMULATION
SKELETON
SPECIFICATIONS
TESTING
XML I/O specification
generative software
input-output programs
parallel I/O solution
parallel computations
parallel processing
producing skeletal I/O applications
skeletal I/O applications