ExaWorks: Workflows for Exascale
- Rutgers Univ., Piscataway, NJ (United States)
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
- Argonne National Lab. (ANL), Argonne, IL (United States); Univ. of Chicago, IL (United States)
- Rutgers Univ., Piscataway, NJ (United States); Brookhaven National Lab. (BNL), Upton, NY (United States)
- Brookhaven National Lab. (BNL), Upton, NY (United States)
- Argonne National Lab. (ANL), Argonne, IL (United States)
Exascale computers will offer transformative capa- bilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific discovery and insight. These software combinations and integra- tions, however, are difficult to achieve due to challenges of coor- dination and deployment of heterogeneous software components on diverse and massive platforms. We present the ExaWorks project, which can address many of these challenges: ExaWorks is leading a co-design process to create a workflow Software Development Toolkit (SDK) consisting of a wide range of work- flow management tools that can be composed and interoperate through common interfaces. We describe the initial set of tools and interfaces supported by the SDK, efforts to make them eas- ier to apply to complex science challenges, and examples of their application to exemplar cases. Furthermore, we discuss how our project is working with the workflows community, large com- puting facilities as well as HPC platform vendors to sustainably address the requirements of workflows at the exascale.
- Research Organization:
- Brookhaven National Laboratory (BNL), Upton, NY (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Basic Energy Sciences (BES)
- DOE Contract Number:
- SC0012704; AC52-07NA27344; AC02-06CH11357
- OSTI ID:
- 1863883
- Report Number(s):
- BNL-222941-2022-JAAM
- Journal Information:
- 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS), Conference: 16th.Workshop on Workflows in Support of Large-Scale Science, St. Louis, MO (United States), 15 Nov 2021
- Country of Publication:
- United States
- Language:
- English
Dataflow coordination of data-parallel tasks via MPI 3.0
|
conference | January 2013 |
High-bypass Learning: Automated Detection of Tumor Cells That Significantly Impact Drug Response
|
conference | November 2020 |
The Exascale Computing Project
|
journal | May 2017 |
funcX: A Federated Function Serving Fabric for Science
|
conference | June 2020 |
CANDLE/Supervisor: a workflow framework for machine learning applied to cancer research
|
journal | December 2018 |
Parsl: Pervasive Parallel Programming in Python
|
conference | January 2019 |
Scalable HPC & AI infrastructure for COVID-19 therapeutics
|
conference | July 2021 |
Compiler Techniques for Massively Scalable Implicit Task Parallelism
|
conference | November 2014 |
SLURM: Simple Linux Utility for Resource Management
|
book | January 2003 |
Turbine: A Distributed-memory Dataflow Engine for High Performance Many-task Applications
|
journal | January 2013 |
Generalizable coordination of large multiscale workflows: challenges and learnings at scale
|
conference | November 2021 |
Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing
|
conference | November 2021 |
A population data-driven workflow for COVID-19 modeling and learning
|
journal | September 2021 |
Similar Records
A Community Roadmap for Scientific Workflows Research and Development
Quantifying Scheduling Challenges for Exascale System Software