Exascale workflow applications and middleware: An ExaWorks retrospective

Alsaadi, Aymen; Hategan-Marandiuc, Mihael; Maheshwari, Ketan; Merzky, Andre; Titov, Mikhail; Turilli, Matteo; Wilke, Andreas; Wozniak, Justin M.; Chard, Kyle; Ferreira da Silva, Rafael; Jha, Shantenu; Laney, Daniel

doi:10.1177/10943420251331674

Exascale workflow applications and middleware: An ExaWorks retrospective

Journal Article · Fri Apr 25 00:00:00 EDT 2025 · International Journal of High Performance Computing Applications

DOI:https://doi.org/10.1177/10943420251331674· OSTI ID:2573590

Alsaadi, Aymen ^[1]; Hategan-Marandiuc, Mihael ^[2]; ^[3]; Merzky, Andre ^[4]; Titov, Mikhail ^[5]; Turilli, Matteo ^[6]; Wilke, Andreas ^[7]; Wozniak, Justin M. ^[2]; Chard, Kyle ^[2]; ^[3]; ^[8]; Laney, Daniel ^[9]

Rutgers Univ., New Brunswick, NJ (United States)
Argonne National Laboratory (ANL), Argonne, IL (United States); Univ. of Chicago, IL (United States)
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Incomputable LLC, Highland Park, NJ (United States)
Brookhaven National Laboratory (BNL), Upton, NY (United States)
Rutgers Univ., New Brunswick, NJ (United States); Brookhaven National Laboratory (BNL), Upton, NY (United States)
Argonne National Laboratory (ANL), Argonne, IL (United States)
Rutgers Univ., New Brunswick, NJ (United States); Princeton Plasma Physics Laboratory (PPPL), Princeton, NJ (United States); Princeton Univ., NJ (United States)
Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Exascale computers offer transformative capabilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific discovery and insight. However, these software combinations and integrations are difficult to achieve due to the challenges of coordinating and deploying heterogeneous software components on diverse and massive platforms. Here, we present the ExaWorks project, which addresses many of these challenges. We developed a workflow Software Development Toolkit (SDK), a curated collection of workflow technologies that can be composed and interoperated through a common interface, engineered following current best practices, and specifically designed to work on HPC platforms. ExaWorks also developed PSI/J, a job management abstraction API, to simplify the construction of portable software components and applications that can be used over various HPC schedulers. The PSI/J API is a minimal interface for submitting and monitoring jobs and their execution state across multiple and commonly used HPC schedulers. We also describe several leading and innovative workflow examples of ExaWorks tools used on DOE leadership platforms. Furthermore, we discuss how our project is working with the workflow community, large computing facilities, and HPC platform vendors to address the requirements of workflows sustainably at the exascale.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Princeton Plasma Physics Laboratory (PPPL), Princeton, NJ (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-06CH11357; AC05-00OR22725; AC52-07NA27344; SC0012704

OSTI ID:: 2573590

Alternate ID(s):: OSTI ID: 2563004
OSTI ID: 2572818
OSTI ID: 2586620

Report Number(s):: BNL--228414-2025-JAAM

Journal Information:: International Journal of High Performance Computing Applications, Journal Name: International Journal of High Performance Computing Applications Journal Issue: 4 Vol. 39; ISSN 1741-2846; ISSN 1094-3420

Publisher:: SAGECopyright Statement

Country of Publication:: United States

Language:: English

References (23)

The INTERSECT Open Federated Architecture for the Laboratory of the Future Engelmann, Christian; Kuchar, Olga; Boehm, Swen Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation, p. 173-190 https://doi.org/10.1007/978-3-031-23606-8_11	book	January 2023
What Is the Price of Simplicity? den Burger, Mathijs; Jacobs, Ceriel; Kielmann, Thilo Euro-Par 2010 - Parallel Processing https://doi.org/10.1007/978-3-642-15277-1_38	book	January 2010
Flux: Overcoming scheduling challenges for exascale workflows Ahn, Dong H.; Bass, Ned; Chu, Albert Future Generation Computer Systems, Vol. 110 https://doi.org/10.1016/j.future.2020.04.006	journal	September 2020
Using Machine Learning at scale in numerical simulations with SmartSim: An application to ocean climate modeling Partee, Sam; Ellis, Matthew; Rigazzi, Alessandro Journal of Computational Science, Vol. 62 https://doi.org/10.1016/j.jocs.2022.101707	journal	July 2022
Swift/T: Large-Scale Application Composition via Distributed-Memory Dataflow Processing Wozniak, J. M.; Armstrong, T. G.; Wilde, M. 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing https://doi.org/10.1109/CCGrid.2013.99	conference	May 2013
Ensemble Toolkit: Scalable and Flexible Execution of Ensembles of Tasks Balasubramanian, Vivekanandan; Treikalis, Antons; Weidner, Ole 2016 45th International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2016.59	conference	August 2016
Flux: A Next-Generation Resource Management Framework for Large HPC Centers Ahn, Dong H.; Garlick, Jim; Grondona, Mark 2014 43nd International Conference on Parallel Processing Workshops (ICCPW), 2014 43rd International Conference on Parallel Processing Workshops https://doi.org/10.1109/ICPPW.2014.15	conference	September 2014
Frontiers in Scientific Workflows: Pervasive Integration With High-Performance Computing Ferreira da Silva, Rafael; Badia, Rosa M.; Bard, Deborah Computer, Vol. 57, Issue 8 https://doi.org/10.1109/MC.2024.3401542	journal	August 2024
Middleware Building Blocks for Workflow Systems Turilli, Matteo; Balasubramanian, Vivek; Merzky, Andre Computing in Science & Engineering, Vol. 21, Issue 4 https://doi.org/10.1109/MCSE.2019.2920048	journal	July 2019
Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing Ward, Logan; Sivaraman, Ganesh; Pauloski, J. Gregory 2021 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC) https://doi.org/10.1109/MLHPC54614.2021.00007	conference	November 2021
Design and Performance Characterization of RADICAL-Pilot on Leadership-Class Platforms Merzky, Andre; Turilli, Matteo; Titov, Mikhail IEEE Transactions on Parallel and Distributed Systems, Vol. 33, Issue 4 https://doi.org/10.1109/TPDS.2021.3105994	journal	April 2022
ExaWorks: Workflows for Exascale Al-Saadi, Aymen; Ahn, Dong H.; Babuji, Yadu 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS54523.2021.00012	conference	November 2021
A Community Roadmap for Scientific Workflows Research and Development da Silva, Rafael Ferreira; Casanova, Henri; Chard, Kyle 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS54523.2021.00016	conference	November 2021
RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms Alsaadi, Aymen; Ward, Logan; Merzky, Andre 2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS56498.2022.00009	conference	November 2022
PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs Hategan-Marandiuc, Mihael; Merzky, Andre; Collier, Nicholson 2023 IEEE 19th International Conference on e-Science (e-Science) https://doi.org/10.1109/e-Science58273.2023.10254912	conference	October 2023
A massively parallel infrastructure for adaptive multiscale simulations: modeling RAS initiation pathway for cancer Di Natale, Francesco; Bhatia, Harsh; Carpenter, Timothy S. SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356197	conference	November 2019
Parsl: Pervasive Parallel Programming in Python Babuji, Yadu; Foster, Ian; Wilde, Michael Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '19 https://doi.org/10.1145/3307681.3325400	conference	January 2019
IMPECCABLE: Integrated Modeling PipelinE for COVID Cure by Assessing Better LEads Saadi, Aymen Al; Alfe, Dario; Babuji, Yadu ICPP 2021: 50th International Conference on Parallel Processing https://doi.org/10.1145/3472456.3473524	conference	October 2021
ExaAM: Metal additive manufacturing simulation at the fidelity of the microstructure Turner, John A.; Belak, James; Barton, Nathan The International Journal of High Performance Computing Applications, Vol. 36, Issue 1 https://doi.org/10.1177/10943420211042558	journal	January 2022
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics Zvyagin, Maxim; Brace, Alexander; Hippe, Kyle The International Journal of High Performance Computing Applications, Vol. 37, Issue 6 https://doi.org/10.1177/10943420231201154	journal	October 2023
Employing artificial intelligence to steer exascale workflows with colmena Ward, Logan; Pauloski, J. Gregory; Hayot-Sasson, Valerie The International Journal of High Performance Computing Applications, Vol. 39, Issue 1 https://doi.org/10.1177/10943420241288242	journal	October 2024
Workflows Community Summit 2022: A Roadmap Revolution Ferreira Da Silva, Rafael; Badia, Rosa; Bala, Venkat https://doi.org/10.2172/2006942	report	March 2023
Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows Ferreira Da Silva, Rafael; Bard, Deborah; Chard, Kyle https://doi.org/10.2172/2474744	report	October 2024

Similar Records

ExaWorks: Workflows for Exascale

Conference · Sun Nov 14 23:00:00 EST 2021 · OSTI ID:1880770

Related Subjects

97 MATHEMATICS AND COMPUTING

Exascale workflow applications and middleware: An ExaWorks retrospective

Citation Formats

References (23)

Similar Records

Related Subjects