Capturing provenance as a diagnostic tool for workflow performance evaluation and optimization
In extreme-scale computing environments such as the DOE Leadership Computing Facilities scientific workflows are routinely used to coordinate software processes for the execution of complex, computational applications that perform in-silico experiments. Monitoring the performance of workflows without also simultaneously tracking provenance is not sufficient to understand variations between runs, configurations, versions of a code, and between changes in an implemented stack, and systems, i.e. the variability of performance metrics data in their historical context. We take a provenance-based approach and demonstrate that provenance is useful as a tool for evaluating and optimizing workflow performance in extreme-scale HPC environments. We present Chimbuko, a framework for the analysis and visualization of the provenance of performance. Chimbuko implements a method for the evaluation of workflow performance from multiple components that enables the exploration of performance metrics data at scale.
- Research Organization:
- Brookhaven National Laboratory (BNL), Upton, NY (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (SC-21); Laboratory-Directed Research and Development (LDRD)
- DOE Contract Number:
- SC0012704
- OSTI ID:
- 1619260
- Report Number(s):
- BNL-209080-2018-PUCP
- Country of Publication:
- United States
- Language:
- English
Similar Records
Prescriptive provenance for streaming analysis of workflows at scale
Computational reproducibility of scientific workflows at extreme scales