Running Ensemble Workflows at Extreme Scale: Lessons Learned and Path Forward

Mehta, Kshitij; Cliff, Ashley; Suter, Fred; Walker, Angelica; Wolf, Matthew; Jacobson, Daniel; Klasky, Scott

doi:10.1109/eScience55777.2022.00042

Running Ensemble Workflows at Extreme Scale: Lessons Learned and Path Forward

Conference · 01 October 2022

DOI:https://doi.org/10.1109/eScience55777.2022.00042· OSTI ID:1968703

^[1]; Cliff, Ashley ^[1]; ^[1]; ^[1]; ^[1]; ^[1]; ^[1]

ORNL

The ever-increasing volumes of scientific data combined with sophisticated techniques for extracting information from them have led to the increasing popularity of ensemble workflows which are a collection of runs of individual workflows. A traditional approach followed by scientists to run ensembles is to rely on simple scripts to execute different runs and manage resources. This approach is not scalable and is error-prone, thereby motivating the development of workflow management systems that specialize in executing ensembles on HPC clusters. However, when the size of both the ensemble and the target system reach extreme scales, existing workflow management systems face new challenges that hamper their efficient execution. In this paper, we describe our experience scaling an ensemble workflow from the computational biology domain from the early design stages to the execution at extreme scale on Summit, a leadership class supercomputer at the Oak Ridge National Laboratory. We discuss challenges that arise when scaling ensembles to several million runs on thousands of HPC nodes. We identify challenges with composition of the ensemble itself, its execution at large scale, post-processing of the generated data, and scalability of the file system. Based on the experience acquired, we develop a generic vision of the capabilities and abstractions to add to existing workflow management systems to enable the execution of ensemble workflows at extreme scales. We believe that the understanding of these fundamental challenges will help application teams along with workflow system developers with designing the next generation of infrastructure for composing and executing extreme-scale ensemble workflows.

View Conference

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1968703

Resource Relation:: Conference: 18th International Conference on e-Science (e-Science 2022) - Salt Lake City, Utah, United States of America - 10/10/2022 9:00:00 AM-10/14/2022 4:00:00 AM

Country of Publication:: United States

Language:: English

References (21)

Parsl: Pervasive Parallel Programming in Python No authors listed Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '19 https://doi.org/10.1145/3307681.3325400		January 2019
Pegasus, a workflow management system for science automation No authors listed Future Generation Computer Systems https://doi.org/10.1016/j.future.2014.10.008		May 2015
Nextflow enables reproducible computational workflows No authors listed Nature Biotechnology https://doi.org/10.1038/nbt.3820		April 2017
ExaWorks: Workflows for Exascale No authors listed 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS54523.2021.00012		November 2021
Swift: A language for distributed parallel scripting No authors listed Parallel Computing https://doi.org/10.1016/j.parco.2011.05.005		September 2011
Workflow Management in Condor No authors listed Workflows for e-Science https://doi.org/10.1007/978-1-84628-757-2_22		January 2007
Using MPI No authors listed MIT Press https://doi.org/10.7551/mitpress/7056.001.0001		January 1999
PMIx: Process management for exascale environments No authors listed Parallel Computing https://doi.org/10.1016/j.parco.2018.08.002		November 2018
ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management No authors listed SoftwareX https://doi.org/10.1016/j.softx.2020.100561		July 2020
An overview of the HDF5 technology suite and its applications No authors listed Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases - AD '11 https://doi.org/10.1145/1966895.1966900		January 2011
A High-Performance Computing Implementation of Iterative Random Forest for the Creation of Predictive Expression Networks No authors listed Genes https://doi.org/10.3390/genes10120996		December 2019
libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations No authors listed IEEE Transactions on Parallel and Distributed Systems https://doi.org/10.1109/TPDS.2021.3082815		April 2022
Iterative random forests to discover predictive and stable high-order interactions No authors listed Proceedings of the National Academy of Sciences https://doi.org/10.1073/pnas.1711236115		January 2018
A Codesign Framework for Online Data Analysis and Reduction No authors listed 2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS49585.2019.00007		November 2019
The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/10943420211019119		May 2021
Feature-preserving Lossy Compression for In Situ Data Analysis No authors listed ICPP Workshops '20: Workshops, 49th International Conference on Parallel Processing - ICPP : Workshops https://doi.org/10.1145/3409390.3409400		August 2020
Serial Generalized Ensemble Simulations of Biomolecules with Self-Consistent Determination of Weights No authors listed Journal of Chemical Theory and Computation https://doi.org/10.1021/ct2008457		February 2012
Generalized-ensemble algorithms: enhanced sampling techniques for Monte Carlo and molecular dynamics simulations No authors listed Journal of Molecular Graphics and Modelling https://doi.org/10.1016/j.jmgm.2003.12.009		May 2004
DeltaFS: exascale file systems scale better without dedicated servers No authors listed Proceedings of the 10th Parallel Data Storage Workshop on - PDSW '15 https://doi.org/10.1145/2834976.2834984		January 2015
Online data analysis and reduction: An important Co-design motif for extreme-scale computers No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/10943420211023549		June 2021
Harnessing the Power of Many: Extensible Toolkit for Scalable Ensemble Applications No authors listed 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2018.00063		May 2018

Similar Records

Toward designing effective exascale scientific computing workflows: experiences and best practices

Conference · 2022 · OSTI ID:1928954

Coletti, Mark; Davidson, Russ; Sedova, Ada

Ensemble Simulations on Leadership Computing Systems

Conference · 2024 · OSTI ID:2538098

Georgiadou, Antigoni; Monge Camacho, Henry; Sohail, Tanvir; +6 more

Workflows Community Summit: Tightening the Integration between Computing Facilities and Scientific Workflows

Technical Report · 2021 · OSTI ID:1842590

Ferreira da Silva, Rafael; Chard, Kyle; Casanova, Henri; +11 more

Running Ensemble Workflows at Extreme Scale: Lessons Learned and Path Forward

Citation Formats

References (21)

Similar Records

Related Subjects