Capturing provenance as a diagnostic tool for workflow performance evaluation and optimization
- Brookhaven National Lab. (BNL), Upton, NY (United States); Brookhaven National Laboratory
- Brookhaven National Lab. (BNL), Upton, NY (United States)
- Stony Brook Univ., NY (United States)
In extreme-scale computing environments such as the DOE Leadership Computing Facilities scientific workflows are routinely used to coordinate software processes for the execution of complex, computational applications that perform in-silico experiments. Monitoring the performance of workflows without also simultaneously tracking provenance is not sufficient to understand variations between runs, configurations, versions of a code, and between changes in an implemented stack, and systems, i.e. the variability of performance metrics data in their historical context. We take a provenance-based approach and demonstrate that provenance is useful as a tool for evaluating and optimizing workflow performance in extreme-scale HPC environments. We present Chimbuko, a framework for the analysis and visualization of the provenance of performance. Chimbuko implements a method for the evaluation of workflow performance from multiple components that enables the exploration of performance metrics data at scale.
- Research Organization:
- Brookhaven National Laboratory
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
- DOE Contract Number:
- SC0012704
- OSTI ID:
- 1556918
- Report Number(s):
- BNL-209080-2018-PUCP; BNL-209080-2018-PUCP; ISBN: 978-1-5386-3161-4
- Journal Information:
- Proceedings, Journal Name: Proceedings
- Country of Publication:
- United States
- Language:
- English
Similar Records
Prescriptive provenance for streaming analysis of workflows at scale
Computational reproducibility of scientific workflows at extreme scales