Workflows Community Summit: Tightening the Integration between Computing Facilities and Scientific Workflows
[1];
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Univ. of Chicago, IL (United States)
- Univ. of Hawaii at Manoa, Honolulu, HI (United States)
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
- Rutgers Univ., New Brunswick, NJ (United States)
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Univ. of Illinois at Urbana-Champaign, IL (United States)
- National Renewable Energy Lab. (NREL), Golden, CO (United States)
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
- Brookhaven National Lab. (BNL), Upton, NY (United States)
- Pittsburgh Supercomputing Center, PA (United States)
Scientific workflows are used almost universally across science domains for solving complex and largescale computing and data analysis problems. The importance of workflows is highlighted by the fact that they have underpinned some of the most significant discoveries of the past decades. Many of these workflows have significant computational, storage, and communication demands, and thus must execute on a range of large-scale computer systems, from local clusters to public clouds and upcoming exascale HPC platforms. Managing these executions is often a significant undertaking, requiring a sophisticated and versatile software infrastructure. Historically, infrastructures for workflow execution consisted of complex, integrated systems, developed in-house by workflow practitioners with strong dependencies on a range of legacy technologies—even including sets of ad hoc scripts. Due to the increasing need to support workflows, dedicated workflow systems were developed to provide abstractions for creating, executing, and adapting workflows conveniently and efficiently while ensuring portability. While these efforts are all worthwhile individually, there are now hundreds of independent workflow systems. These workflow systems are created and used by thousands of researchers and developers, leading to a rapidly growing corpus of workflows research publications. The resulting workflow system technology landscape is fragmented, which may present significant barriers for future workflow users due to many seemingly comparable, yet usually mutually incompatible, systems that exist. In order to tackle some of the challenges described above, the DOE-funded ExaWorks and NSF-funded WorkflowsRI projects have organized in 2021 a series of events entitled the “Workflows Community Summit”. The third edition of the “Workflows Community Summit” explored workflows challenges and opportunities from the perspective of computing centers and facilities. This third summit builds on two prior summits (https://workflowsri.org/summits) that (i) established a high level vision for workflows research; and (ii) explored technical approaches for realizing that vision. The third summit brought together a small group of facilities representatives with the aim to understand how workflows are currently being used at each facility, how facilities would like to interact with workflow developers and users, how workflows fit with facility roadmaps, and what opportunities there are for tighter integration between facilities and workflows. This report documents and organizes the wealth of information provided by the participants before, during, and after the summit.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1842590
- Report Number(s):
- ORNL/TM-2022/1832
- Resource Relation:
- Conference: Workflows Community Summit: Tightening the Integration between Computing Facilities and Scientific Workflows, Held Virtually (United States), 8 Nov 2021; Related Information: https://workflowsri.org/summits/
- Country of Publication:
- United States
- Language:
- English
Similar Records
WE-A-16A-01: International Medical Physics Symposium: Increasing Access to Medical Physics Education/Training and Research Excellence
ESnet Requirements Review Program Through the IRI Lens: A Meta-Analysis of Workflow Patterns Across DOE Office of Science Programs (Final Report)