Computational reproducibility of scientific workflows at extreme scales

Pouchard, Line; Baldwin, Sterling; Elsethagen, Todd; Jha, Shantenu; Raju, Bibi; Stephan, Eric; Tang, Li; Van Dam, Kerstin Kleese

doi:10.1177/1094342019839124

Title: Computational reproducibility of scientific workflows at extreme scales

Journal Article · Mon Apr 08 00:00:00 EDT 2019 · International Journal of High Performance Computing Applications

DOI:https://doi.org/10.1177/1094342019839124· OSTI ID:1542776

^[1]; Baldwin, Sterling ^[2]; Elsethagen, Todd ^[3]; Jha, Shantenu ^[1]; Raju, Bibi ^[3]; Stephan, Eric ^[3]; Tang, Li ^[4]; Van Dam, Kerstin Kleese ^[1]

Brookhaven National Lab. (BNL), Upton, NY (United States)
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics. We discuss two use cases: scientific reproducibility of results in the Energy Exascale Earth System Model (E3SM – previously ACME), and performance reproducibility in molecular dynamics workflows on HPC computing platforms. In order to capture and persist the provenance and performance data of these workflows, we have designed and developed the Chimbuko and ProvEn frameworks. Chimbuko captures provenance and enables detailed single workflow performance analysis. ProvEn is a hybrid, queriable system for storing and analyzing the provenance and performance metrics of multiple runs in workflow performance analysis campaigns. Workflow provenance and performance data output from Chimbuko can be visualized in a dynamic, multi-level visualization providing overview and zoom-in capabilities for areas of interest. Provenance and related performance data ingested into ProvEn is queriable and can be used to reproduce runs. In conclusion, our provenance-based approach highlights challenges in extracting information and gaps in the information collected. It is agnostic to the type of provenance data it captures so that both the reproducibility of scientific results and that of performance can be explored with our tools.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (SC-21)

Grant/Contract Number:: SC0012704

OSTI ID:: 1542776

Report Number(s):: BNL-211854-2019-JAAM

Journal Information:: International Journal of High Performance Computing Applications, Vol. 33, Issue 5; ISSN 1094-3420

Publisher:: SAGECopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 6 works

Citation information provided by
Web of Science

References (29)

An introduction to Docker for reproducible research Boettiger, Carl ACM SIGOPS Operating Systems Review, Vol. 49, Issue 1 https://doi.org/10.1145/2723872.2723882	journal	January 2015
Provenance: An Introduction to PROV Moreau, Luc; Groth, Paul Synthesis Lectures on the Semantic Web: Theory and Technology, Vol. 3, Issue 4 https://doi.org/10.2200/S00528ED1V01Y201308WBE007	journal	September 2013
Leveraging large sensor streams for robust cloud control Singh, Alok; Stephan, Eric; Elsethagen, Todd 2016 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2016.7840839	conference	December 2016
A web interface for XALT log data analysis Huang, Ruizhu; Xu, Weijia; McLay, Robert Proceedings of the XSEDE16 on Diversity, Big Data, and Science at Scale - XSEDE16 https://doi.org/10.1145/2949550.2949560	conference	January 2016
Packing experiments for sharing and publication Chirigati, Fernando; Shasha, Dennis; Freire, Juliana Proceedings of the 2013 international conference on Management of data - SIGMOD '13 https://doi.org/10.1145/2463676.2465269	conference	January 2013
Ten Simple Rules for Reproducible Computational Research Sandve, Geir Kjetil; Nekrutenko, Anton; Taylor, James PLoS Computational Biology, Vol. 9, Issue 10 https://doi.org/10.1371/journal.pcbi.1003285	journal	October 2013
Performance Visualization for TAU Instrumented Scientific Workflows [Performance Visualization for TAU Instrumented Scientific Workflows] Xie, Cong; Xu, Wei; Ha, Sungsoo International Conference on Information Visualization Theory and Applications, Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications https://doi.org/10.5220/0006646803330340	conference	January 2018
Towards trustworthy testbeds thanks to throughout testing Nussbaum, Lucas 2017 IEEE International Parallel and Distributed Processing Symposium: Workshops (IPDPSW), 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) https://doi.org/10.1109/IPDPSW.2017.101	conference	May 2017
Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks: HELLO ADIOS Liu, Qing; Logan, Jeremy; Tian, Yuan Concurrency and Computation: Practice and Experience, Vol. 26, Issue 7 https://doi.org/10.1002/cpe.3125	journal	August 2013
Numerical reproducibility for the parallel reduction on multi- and many-core architectures Collange, Sylvain; Defour, David; Graillat, Stef Parallel Computing, Vol. 49 https://doi.org/10.1016/j.parco.2015.09.001	journal	November 2015
High Performance MPI Library for Container-Based HPC Cloud on InfiniBand Clusters Zhang, Jie; Lu, Xiaoyi; Panda, Dhabaleswar K. 2016 45th International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2016.38	conference	August 2016
Extreme Heterogeneity 2018 - Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity Vetter, Jeffrey S.; Brightwell, Ron; Gokhale, Maya https://doi.org/10.2172/1473756	report	December 2018
Fast Parallel Algorithms for Short-Range Molecular Dynamics Plimpton, Steve Journal of Computational Physics, Vol. 117, Issue 1 https://doi.org/10.1006/jcph.1995.1039	journal	March 1995
Prescriptive provenance for streaming analysis of workflows at scale Pouchard, Line; Huck, Kevin; Matyasfalvi, Gyorgy 2018 New York Scientific Data Summit (NYSDS) https://doi.org/10.1109/NYSDS.2018.8538951	conference	August 2018
User Environment Tracking and Problem Detection with XALT Agrawal, Kapil; Fahey, Mark R.; McLay, Robert 2014 First International Workshop on HPC User Support Tools (HUST) https://doi.org/10.1109/HUST.2014.6	conference	November 2014
Guidelines for evaluating and expressing the uncertainty of NIST measurement results Taylor, B. N. NIST Technical Note 1297 1994 Edition https://doi.org/10.6028/NIST.TN.1297	report	January 1994
The Spack package manager: bringing order to HPC software chaos Gamblin, Todd; LeGendre, Matthew; Collette, Michael R. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807623	conference	January 2015
R3: repeatability, reproducibility and rigor Vitek, Jan; Kalibera, Tomas ACM SIGPLAN Notices, Vol. 47, Issue 4a https://doi.org/10.1145/2442776.2442781	journal	March 2012
Data provenance hybridization supporting extreme-scale scientific workflow applications Elsethagen, Todd; Stephan, Eric; Raju, Bibi 2016 New York Scientific Data Summit (NYSDS) https://doi.org/10.1109/NYSDS.2016.7747819	conference	August 2016
Streaming spectral clustering Yoo, Shinjae; Huang, Hao; Kasiviswanathan, Shiva Prasad 2016 IEEE 32nd International Conference on Data Engineering (ICDE) https://doi.org/10.1109/ICDE.2016.7498277	conference	May 2016
Toward the Geoscience Paper of the Future: Best practices for documenting and sharing research from data to software to provenance Gil, Yolanda; David, Cédric H.; Demir, Ibrahim Earth and Space Science, Vol. 3, Issue 10 https://doi.org/10.1002/2015EA000136	journal	October 2016
Enhancing reproducibility for computational methods Stodden, V.; McNutt, M.; Bailey, D. H. Science, Vol. 354, Issue 6317 https://doi.org/10.1126/science.aah6168	journal	December 2016
Reproducible Research in Computational Science Peng, R. D. Science, Vol. 334, Issue 6060 https://doi.org/10.1126/science.1213847	journal	December 2011
Data storage and sharing for the long tail of science Zhang, Boyu; Pouchard, Line C.; Smith, Preston M. 2016 New York Scientific Data Summit (NYSDS) https://doi.org/10.1109/NYSDS.2016.7747811	conference	August 2016
Numerical Reproducibility and Accuracy at ExaScale Demmel, J. 2013 IEEE 21st Symposium on Computer Arithmetic (ARITH) https://doi.org/10.1109/ARITH.2013.43	conference	April 2013
Testbeds Support for Reproducible Research Nussbaum, Lucas Proceedings of the Reproducibility Workshop on ZZZ - Reproducibility '17 https://doi.org/10.1145/3097766.3097773	conference	January 2017
Capturing provenance as a diagnostic tool for workflow performance evaluation and optimization Pouchard, Line; Malik, Abid; Dam, Huub Van 2017 New York Scientific Data Summit (NYSDS) https://doi.org/10.1109/NYSDS.2017.8085043	conference	August 2017
The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data Cinquini, Luca; Crichton, Daniel; Mattmann, Chris Future Generation Computer Systems, Vol. 36 https://doi.org/10.1016/j.future.2013.07.002	journal	July 2014
Automated Capture of Experiment Context for Easier Reproducibility in Computational Research Davison, Andrew Computing in Science & Engineering, Vol. 14, Issue 4 https://doi.org/10.1109/MCSE.2012.41	journal	July 2012

Similar Records

Computational Reproducibility of Scientific Workflows at Extreme Scales

Journal Article · Sun Sep 01 00:00:00 EDT 2019 · International Journal of High Performance Computing Applications · OSTI ID:1542776

Pouchard, Line C.; Baldwin, Sterling A.; Elsethagen, Todd O.; +5 more

Capturing provenance as a diagnostic tool for workflow performance evaluation and optimization

Conference · Sun Aug 06 00:00:00 EDT 2017 · OSTI ID:1542776

Pouchard, Line

Capturing provenance as a diagnostic tool for workflow performance evaluation and optimization

Conference · Wed Aug 30 00:00:00 EDT 2017 · Proceedings · OSTI ID:1542776

Pouchard, Line; Malik, Abid; Van Dam, Huub; +3 more

Related Subjects

97 MATHEMATICS AND COMPUTING
computational reproducibility
scientific workflows
provenance
performance analysis
ProvEn
Chimbuko

Title: Computational reproducibility of scientific workflows at extreme scales

Citation Formats

References (29)

Similar Records

Related Subjects