Extreme-scale workflows: A perspective from the JLESC international community

Yildiz, Orcun; Gueroudji, Amal; Bigot, Julien; Raffin, Bruno; Badia, Rosa M.; Peterka, Tom

doi:10.1016/j.future.2024.07.041

Extreme-scale workflows: A perspective from the JLESC international community

Journal Article · Tue Jul 23 00:00:00 EDT 2024 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2024.07.041· OSTI ID:2440426

Yildiz, Orcun ^[1]; Gueroudji, Amal ^[1]; Bigot, Julien ^[2]; Raffin, Bruno ^[3]; Badia, Rosa M. ^[4]; Peterka, Tom ^[1]

Argonne National Laboratory (ANL), Argonne, IL (United States)
Univ. Paris-Saclay, Gif-sur-Yvette (France); Centre National de la Recherche Scientifique (CNRS) (France)
Univ. of Grenoble Alpes, Grenoble (France); Centre National de la Recherche Scientifique (CNRS) (France)
Barcelona Supercomputing Center (Spain)

The Joint Laboratory for Extreme-Scale Computing (JLESC) focuses on software challenges in high-performance computing systems to meet the needs of today’s science campaigns, which often require large resources, consist of multiple tasks, and generate vast amounts of data. In this context, extreme-scale workflows have been the key factor in enabling scientific discoveries by helping scientists automate the dependencies and data exchanges between workflow tasks, instead of managing those manually. Here, in this paper, we present representative extreme-scale workflows and feature workflow systems developed by JLESC participating institutions. We present lessons learned while developing these tools, alongside with the open challenges and future research directions in the field of extreme-scale workflows.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 2440426

Alternate ID(s):: OSTI ID: 2427007

Journal Information:: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Vol. 161; ISSN 0167-739X

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (19)

DataSpaces: an interaction and coordination framework for coupled simulation workflows Docan, Ciprian; Parashar, Manish; Klasky, Scott Cluster Computing, Vol. 15, Issue 2 https://doi.org/10.1007/s10586-011-0162-y	journal	February 2011
Workflows and e-Science: An overview of workflow system features and capabilities Deelman, Ewa; Gannon, Dennis; Shields, Matthew Future Generation Computer Systems, Vol. 25, Issue 5 https://doi.org/10.1016/j.future.2008.06.012	journal	May 2009
Efficient development of high performance data analytics in Python Álvarez Cid-Fuentes, Javier; Álvarez, Pol; Amela, Ramon Future Generation Computer Systems, Vol. 111 https://doi.org/10.1016/j.future.2019.09.051	journal	October 2020
Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence Ejarque, Jorge; Badia, Rosa M.; Albertin, Loïc Future Generation Computer Systems, Vol. 134 https://doi.org/10.1016/j.future.2022.04.014	journal	September 2022
Dataclay: A distributed data store for effective inter-player data sharing Martí, Jonathan; Queralt, Anna; Gasull, Daniel Journal of Systems and Software, Vol. 131 https://doi.org/10.1016/j.jss.2017.05.080	journal	September 2017
Nextflow enables reproducible computational workflows Di Tommaso, Paolo; Chatzou, Maria; Floden, Evan W. Nature Biotechnology, Vol. 35, Issue 4 https://doi.org/10.1038/nbt.3820	journal	April 2017
Nyx: A MASSIVELY PARALLEL AMR CODE FOR COMPUTATIONAL COSMOLOGY Almgren, Ann S.; Bell, John B.; Lijewski, Mike J. The Astrophysical Journal, Vol. 765, Issue 1 https://doi.org/10.1088/0004-637X/765/1/39	journal	February 2013
The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2020 update Jalili, Vahid; Afgan, Enis; Gu, Qiang Nucleic Acids Research, Vol. 48, Issue W1 https://doi.org/10.1093/nar/gkaa434	journal	June 2020
LowFive: In Situ Data Transport for High-Performance Workflows Peterka, Tom; Morozov, Dmitriy; Nigmetov, Arnur 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS54959.2023.00102	conference	May 2023
Heterogeneous Hierarchical Workflow Composition Yildiz, Orcun; Ejarque, Jorge; Chan, Henry Computing in Science & Engineering, Vol. 21, Issue 4 https://doi.org/10.1109/MCSE.2019.2918766	journal	July 2019
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) Lofstead, Jay F.; Klasky, Scott; Schwan, Karsten Proceedings of the 6th international workshop on Challenges of large applications in distributed environments - CLADE '08 https://doi.org/10.1145/1383529.1383533	conference	January 2008
An overview of the HDF5 technology suite and its applications Folk, Mike; Heber, Gerd; Koziol, Quincey Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases - AD '11 https://doi.org/10.1145/1966895.1966900	conference	January 2011
Damaris: Addressing Performance Variability in Data Management for Post-Petascale Simulations Dorier, Matthieu; Antoniu, Gabriel; Cappello, Franck ACM Transactions on Parallel Computing, Vol. 3, Issue 3 https://doi.org/10.1145/2987371	journal	October 2016
The challenges of elastic in situ analysis and visualization Dorier, Matthieu; Yildiz, Orcun; Peterka, Tom Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization https://doi.org/10.1145/3364228.3364234	conference	November 2019
funcX: A Federated Function Serving Fabric for Science Chard, Ryan; Babuji, Yadu; Li, Zhuozhao HPDC '20: The 29th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3369583.3392683	conference	June 2020
PyCOMPSs: Parallel computational workflows in Python Tejedor, Enric; Becerra, Yolanda; Alomar, Guillem The International Journal of High Performance Computing Applications, Vol. 31, Issue 1 https://doi.org/10.1177/1094342015594678	journal	July 2016
In situ and in-transit analysis of cosmological simulations Friesen, Brian; Almgren, Ann; Lukić, Zarija Computational Astrophysics and Cosmology, Vol. 3, Issue 1 https://doi.org/10.1186/s40668-016-0017-2	journal	August 2016
Melissa: coordinating large-scale ensemble runs for deep learning and sensitivity analyses Schouler, Marc; Caulk, Robert Alexander; Meyer, Lucas Journal of Open Source Software, Vol. 8, Issue 86 https://doi.org/10.21105/joss.05291	journal	June 2023
Dask: Parallel Computation with Blocked algorithms and Task Scheduling Rocklin, Matthew Proceedings of the Python in Science Conference https://doi.org/10.25080/Majora-7b98e3ed-013	conference	January 2015

Similar Records

A characterization of workflow management systems for extreme-scale applications

Journal Article · Wed Feb 15 19:00:00 EST 2017 · Future Generations Computer Systems · OSTI ID:1408072

PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows

Journal Article · Mon Jul 13 20:00:00 EDT 2015 · International Journal of High Performance Computing Applications · OSTI ID:1265426

The future of scientific workflows

Journal Article · Tue Apr 25 20:00:00 EDT 2017 · International Journal of High Performance Computing Applications · OSTI ID:1422587

Related Subjects

97 MATHEMATICS AND COMPUTING
Data model
Data transport
Extreme-scale workflows
HPC
In situ

Extreme-scale workflows: A perspective from the JLESC international community

Citation Formats

References (19)

Similar Records

Related Subjects