skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Lightweight Provenance Service for High-Performance Computing

Abstract

Provenance describes detailed information about the history of a piece of data, containing the relationships among elements such as users, processes, jobs, and workflows that contribute to the existence of data. Provenance is key to supporting many data management functionalities that are increasingly important in operations such as identifying data sources, parameters, or assumptions behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. Despite its importance, however, provenance support is largely underdeveloped in highly parallel architectures and systems. One major challenge is the demanding requirements of providing provenance service in situ. The need to remain lightweight and to be always on often conflicts with the need to be transparent and offer an accurate catalog of details regarding the applications and systems. To tackle this challenge, we introduce a lightweight provenance service, called LPS, for high-performance computing (HPC) systems. LPS leverages a kernel instrument mechanism to achieve transparency and introduces representative execution and flexible granularity to capture comprehensive provenance with controllable overhead. Extensive evaluations and use cases have confirmed its efficiency and usability. We believe that LPS can be integrated into current and future HPC systems to support a variety of data managementmore » needs.« less

Authors:
; ; ; ;
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22); National Science Foundation (NSF)
OSTI Identifier:
1392595
DOE Contract Number:  
AC02-06CH11357
Resource Type:
Conference
Resource Relation:
Conference: The 26th International Conference on Parallel Architectures and Compilation Techniques (PACT), 09/09/17 - 09/13/17, Portland, OR, USA
Country of Publication:
United States
Language:
English

Citation Formats

Dai, Dong, Chen, Yong, Carns, Philip, Jenkins, John, and Ross, Robert. Lightweight Provenance Service for High-Performance Computing. United States: N. p., 2017. Web.
Dai, Dong, Chen, Yong, Carns, Philip, Jenkins, John, & Ross, Robert. Lightweight Provenance Service for High-Performance Computing. United States.
Dai, Dong, Chen, Yong, Carns, Philip, Jenkins, John, and Ross, Robert. Sat . "Lightweight Provenance Service for High-Performance Computing". United States. https://www.osti.gov/servlets/purl/1392595.
@article{osti_1392595,
title = {Lightweight Provenance Service for High-Performance Computing},
author = {Dai, Dong and Chen, Yong and Carns, Philip and Jenkins, John and Ross, Robert},
abstractNote = {Provenance describes detailed information about the history of a piece of data, containing the relationships among elements such as users, processes, jobs, and workflows that contribute to the existence of data. Provenance is key to supporting many data management functionalities that are increasingly important in operations such as identifying data sources, parameters, or assumptions behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. Despite its importance, however, provenance support is largely underdeveloped in highly parallel architectures and systems. One major challenge is the demanding requirements of providing provenance service in situ. The need to remain lightweight and to be always on often conflicts with the need to be transparent and offer an accurate catalog of details regarding the applications and systems. To tackle this challenge, we introduce a lightweight provenance service, called LPS, for high-performance computing (HPC) systems. LPS leverages a kernel instrument mechanism to achieve transparency and introduces representative execution and flexible granularity to capture comprehensive provenance with controllable overhead. Extensive evaluations and use cases have confirmed its efficiency and usability. We believe that LPS can be integrated into current and future HPC systems to support a variety of data management needs.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2017},
month = {9}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: