skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: An Optimizing Compiler for Petascale I/O on Leadership-Class Architectures

Abstract

In high-performance computing (HPC), parallel I/O architectures usually have very complex hierarchies with multiple layers that collectively constitute an I/O stack, including high-level I/O libraries such as PnetCDF and HDF5, I/O middleware such as MPI-IO, and parallel file systems such as PVFS and Lustre. Our DOE project explored automated instrumentation and compiler support for I/O intensive applications. Our project made significant progress towards understanding the complex I/O hierarchies of high-performance storage systems (including storage caches, HDDs, and SSDs), and designing and implementing state-of-the-art compiler/runtime system technology that targets I/O intensive HPC applications that target leadership class machine. This final report summarizes the major achievements of the project and also points out promising future directions Two new sections in this report compared to the previous report are IOGenie and SSD/NVM-specific optimizations.

Authors:
 [1];  [2];  [3]
+ Show Author Affiliations
  1. PSU
  2. Northwestern
  3. ANL
Publication Date:
Research Org.:
The Pennsylvania State University
Sponsoring Org.:
USDOE
OSTI Identifier:
1123486
Report Number(s):
DOE-PSU-02156
DOE Contract Number:
SC0002156
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Kandemir, Mahmut Taylan, Choudary, Alok, and Thakur, Rajeev. An Optimizing Compiler for Petascale I/O on Leadership-Class Architectures. United States: N. p., 2014. Web. doi:10.2172/1123486.
Copy to clipboard
Kandemir, Mahmut Taylan, Choudary, Alok, & Thakur, Rajeev. An Optimizing Compiler for Petascale I/O on Leadership-Class Architectures. United States. doi:10.2172/1123486.
Copy to clipboard
Kandemir, Mahmut Taylan, Choudary, Alok, and Thakur, Rajeev. Sat . "An Optimizing Compiler for Petascale I/O on Leadership-Class Architectures". United States. doi:10.2172/1123486. https://www.osti.gov/servlets/purl/1123486.
Copy to clipboard
@article{osti_1123486,
title = {An Optimizing Compiler for Petascale I/O on Leadership-Class Architectures},
author = {Kandemir, Mahmut Taylan and Choudary, Alok and Thakur, Rajeev},
abstractNote = {In high-performance computing (HPC), parallel I/O architectures usually have very complex hierarchies with multiple layers that collectively constitute an I/O stack, including high-level I/O libraries such as PnetCDF and HDF5, I/O middleware such as MPI-IO, and parallel file systems such as PVFS and Lustre. Our DOE project explored automated instrumentation and compiler support for I/O intensive applications. Our project made significant progress towards understanding the complex I/O hierarchies of high-performance storage systems (including storage caches, HDDs, and SSDs), and designing and implementing state-of-the-art compiler/runtime system technology that targets I/O intensive HPC applications that target leadership class machine. This final report summarizes the major achievements of the project and also points out promising future directions Two new sections in this report compared to the previous report are IOGenie and SSD/NVM-specific optimizations.},
doi = {10.2172/1123486},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Sat Mar 01 00:00:00 EST 2014},
month = {Sat Mar 01 00:00:00 EST 2014}
}
Copy to clipboard
Technical Report:
View Technical Report (1.61 MB)
DOI:  10.2172/1123486
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

  • ;
    In high-performance computing systems, parallel I/O architectures usually have very complex hierarchies with multiple layers that collectively constitute an I/O stack, including high-level I/O libraries such as PnetCDF and HDF5, I/O middleware such as MPI-IO, and parallel file systems such as PVFS and Lustre. Our project explored automated instrumentation and compiler support for I/O intensive applications. Our project made significant progress towards understanding the complex I/O hierarchies of high-performance storage systems (including storage caches, HDDs, and SSDs), and designing and implementing state-of-the-art compiler/runtime system technology that targets I/O intensive HPC applications that target leadership class machine. This final report summarizesmore » the major achievements of the project and also points out promising future directions.« less

  • OSC is a compiler and runtime system for the functional language Sisal. Functional languages are based on mathematical principals, and may reduce the cost of parallel program development without sacrificing performance. OSC compiles Sisal source code to binary form, automatically inserting calls to the Sisal runtime system to manage parallel execution of independent tasks. Features include support for dynamic arrays, automatic vectorization, and automatic parallelization. At runtime, the user may specify the number of workers, the granularity of tasks, and other execution parameters.

  • OSC is a compiler and runtime system for the functional language Sisal. Functional languages are based on mathematical principals, and may reduce the cost of parallel program development without sacrificing performance. OSC compiles Sisal source code to binary form, automatically inserting calls to the Sisal runtime system to manage parallel execution of independent tasks. Features include support for dynamic arrays, automatic vectorization, and automatic parallelization. At runtime, the user may specify the number of workers, the granularity of tasks, and other execution parameters.

  • Computational scientists must understand results from experimental, observational and computational simulation generated data to gain insights and perform knowledge discovery. As systems approach the petascale range, problems that were unimaginable a few years ago are within reach. With the increasing volume and complexity of data produced by ultra-scale simulations and high-throughput experiments, understanding the science is largely hampered by the lack of comprehensive I/O, storage, acceleration of data manipulation, analysis, and mining tools. Scientists require techniques, tools and infrastructure to facilitate better understanding of their data, in particular the ability to effectively perform complex data analysis, statistical analysis and knowledgemore » discovery. The goal of this work is to enable more effective analysis of scientific datasets through the integration of enhancements in the I/O stack, from active storage support at the file system layer to MPI-IO and high-level I/O library layers. We propose to provide software components to accelerate data analytics, mining, I/O, and knowledge discovery for large-scale scientific applications, thereby increasing productivity of both scientists and the systems. Our approaches include 1) design the interfaces in high-level I/O libraries, such as parallel netCDF, for applications to activate data mining operations at the lower I/O layers; 2) Enhance MPI-IO runtime systems to incorporate the functionality developed as a part of the runtime system design; 3) Develop parallel data mining programs as part of runtime library for server-side file system in PVFS file system; and 4) Prototype an active storage cluster, which will utilize multicore CPUs, GPUs, and FPGAs to carry out the data mining workload.« less

  • This document is the final report for a multi-year effort building infrastructure to support tool development for Fortran programs. We also investigated static analysis and code transformation methods relevant to scientific programmers who are writing Fortran programs for petascale-class high performance computing systems. This report details our accomplishments, technical approaches, and provides information on where the research results and code may be obtained from an open source software repository. The report for the first year of the project that was performed at the University of Oregon prior to the PI moving to Galois, Inc. is included as an appendix.