A characterization of workflow management systems for extreme-scale applications

Ferreira da Silva, Rafael; Filgueira, Rosa; Pietri, Ilia; Jiang, Ming; Sakellariou, Rizos; Deelman, Ewa

doi:10.1016/j.future.2017.02.026

A characterization of workflow management systems for extreme-scale applications

Journal Article · Thu Feb 16 00:00:00 EST 2017 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2017.02.026· OSTI ID:1408072

Ferreira da Silva, Rafael ^[1]; Filgueira, Rosa ^[2]; Pietri, Ilia ^[3]; Jiang, Ming ^[4]; Sakellariou, Rizos ^[5]; Deelman, Ewa ^[1]

University of Southern California, Marina del Rey, CA (United States)
British Geological Survey, Lyell Centre, Edinburgh (United Kingdom); University of Edinburgh (United Kingdom). School of Informatics
Univ. of Athens (Greece). Department of Informatics and Telecommunication
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Univ. of Manchester (United Kingdom). School of Computer Science

We present that the automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today’s computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. Finally, the paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC52-07NA27344; SC0012636

OSTI ID:: 1408072

Alternate ID(s):: OSTI ID: 1495625

Report Number(s):: LLNL-JRNL--706700

Journal Information:: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Journal Issue: C Vol. 75; ISSN 0167-739X

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (51)

Galaxy: A Web‐Based Genome Analysis Tool for Experimentalists Blankenberg, Daniel; Kuster, Gregory Von; Coraor, Nathaniel Current Protocols in Molecular Biology, Vol. 89, Issue 1 https://doi.org/10.1002/0471142727.mb1910s89	journal	January 2010
FireWorks: a dynamic workflow system designed for high-throughput applications: FireWorks: A Dynamic Workflow System Designed for High-Throughput Applications Jain, Anubhav; Ong, Shyue Ping; Chen, Wei Concurrency and Computation: Practice and Experience, Vol. 27, Issue 17 https://doi.org/10.1002/cpe.3505	journal	May 2015
A Taxonomy of Workflow Management Systems for Grid Computing Yu, Jia; Buyya, Rajkumar Journal of Grid Computing, Vol. 3, Issue 3-4 https://doi.org/10.1007/s10723-005-9010-8	journal	September 2005
A Survey of Data-Intensive Scientific Workflow Management Liu, Ji; Pacitti, Esther; Valduriez, Patrick Journal of Grid Computing, Vol. 13, Issue 4 https://doi.org/10.1007/s10723-015-9329-8	journal	March 2015
Workflows and extensions to the Kepler scientific workflow system to support environmental sensor data access and analysis Barseghian, Derik; Altintas, Ilkay; Jones, Matthew B. Ecological Informatics, Vol. 5, Issue 1 https://doi.org/10.1016/j.ecoinf.2009.08.008	journal	January 2010
Workflows and e-Science: An overview of workflow system features and capabilities Deelman, Ewa; Gannon, Dennis; Shields, Matthew Future Generation Computer Systems, Vol. 25, Issue 5 https://doi.org/10.1016/j.future.2008.06.012	journal	May 2009
Characterizing and profiling scientific workflows Juve, Gideon; Chervenak, Ann; Deelman, Ewa Future Generation Computer Systems, Vol. 29, Issue 3 https://doi.org/10.1016/j.future.2012.08.015	journal	March 2013
Pegasus, a workflow management system for science automation Deelman, Ewa; Vahi, Karan; Juve, Gideon Future Generation Computer Systems, Vol. 46 https://doi.org/10.1016/j.future.2014.10.008	journal	May 2015
Algorithms for cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds Malawski, Maciej; Juve, Gideon; Deelman, Ewa Future Generation Computer Systems, Vol. 48 https://doi.org/10.1016/j.future.2015.01.004	journal	July 2015
Self-scalable services in service oriented software for cost-effective data farming Król, Dariusz; Kitowski, Jacek Future Generation Computer Systems, Vol. 54 https://doi.org/10.1016/j.future.2015.07.003	journal	January 2016
Swift: A language for distributed parallel scripting Wilde, Michael; Hategan, Mihael; Wozniak, Justin M. Parallel Computing, Vol. 37, Issue 9 https://doi.org/10.1016/j.parco.2011.05.005	journal	September 2011
The Spallation Neutron Source in Oak Ridge: A powerful tool for materials research Mason, T. E.; Abernathy, D.; Anderson, I. Physica B: Condensed Matter, Vol. 385-386 https://doi.org/10.1016/j.physb.2006.05.281	journal	November 2006
Using a suite of ontologies for preserving workflow-centric research objects Belhajjame, Khalid; Zhao, Jun; Garijo, Daniel Journal of Web Semantics, Vol. 32 https://doi.org/10.1016/j.websem.2015.01.003	journal	May 2015
Taverna: a tool for the composition and enactment of bioinformatics workflows Oinn, T.; Addis, M.; Ferris, J. Bioinformatics, Vol. 20, Issue 17 https://doi.org/10.1093/bioinformatics/bth361	journal	June 2004
Taverna: a tool for building and running workflows of services Hull, D.; Wolstencroft, K.; Stevens, R. Nucleic Acids Research, Vol. 34, Issue Web Server https://doi.org/10.1093/nar/gkl320	journal	July 2006
The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud Wolstencroft, Katherine; Haines, Robert; Fellows, Donal Nucleic Acids Research, Vol. 41, Issue W1 https://doi.org/10.1093/nar/gkt328	journal	May 2013
SciCumulus: A Lightweight Cloud Middleware to Explore Many Task Computing Paradigm in Scientific Workflows de Oliveira, Daniel; Ogasawara, Eduardo; Baião, Fernanda 2010 IEEE International Conference on Cloud Computing (CLOUD), 2010 IEEE 3rd International Conference on Cloud Computing https://doi.org/10.1109/CLOUD.2010.64	conference	July 2010
Practical Resource Monitoring for Robust High Throughput Computing Juve, Gideon; Tovar, Benjamin; Silva, Rafael Ferreira da 2015 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2015.115	conference	September 2015
A Steering Environment for Online Parallel Visualization of Legacy Parallel Simulations Esnard, Aurelien; Richart, Nicolas; Coulaud, Olivier Proceedings. Tenth IEEE International Symposium on Distributed Simulation and Real-Time Applications, 2006 Tenth IEEE International Symposium on Distributed Simulation and Real-Time Applications https://doi.org/10.1109/DS-RT.2006.7	conference	October 2006
Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-Intensive Science Filgueira, Rosa; Da Silva, Rafael Ferreira; Krause, Amrey 2016 Seventh International Workshop on Data-Intensive Computing in the Clouds (DataCloud) https://doi.org/10.1109/DataCloud.2016.004	conference	November 2016
Science automation in practice: Performance data farming in workflows Krol, Dariusz; Kitowski, Jacek; da Silva, Rafael Ferreira 2016 IEEE 21st International Conference on Emerging Technologies and Factory Automation (ETFA) https://doi.org/10.1109/ETFA.2016.7733677	conference	September 2016
A Supervised Learning Framework for Arbitrary Lagrangian-Eulerian Simulations Jiang, Ming; Gallagher, Brian; Kallman, Joshua 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA) https://doi.org/10.1109/ICMLA.2016.0176	conference	December 2016
Adaptable, metadata rich IO methods for portable high performance IO Lofstead, Jay; Zheng, Fang; Klasky, Scott Distributed Processing (IPDPS), 2009 IEEE International Symposium on Parallel & Distributed Processing https://doi.org/10.1109/IPDPS.2009.5161052	conference	May 2009
Enabling In-situ Execution of Coupled Scientific Workflow on Multi-core Platform Zhang, Fan; Docan, Ciprian; Parashar, Manish 2012 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), 2012 IEEE 26th International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2012.122	conference	May 2012
Toward simulation-time data analysis and I/O acceleration on leadership-class systems Vishwanath, Venkatram; Hereld, Mark; Papka, Michael E. 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092178	conference	October 2011
The ParaView Coprocessing Library: A scalable, general purpose in situ visualization library Fabian, Nathan; Moreland, Kenneth; Thompson, David 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092322	conference	October 2011
In Situ Visualization at Extreme Scale: Challenges and Opportunities No authors listed IEEE Computer Graphics and Applications, Vol. 29, Issue 6 https://doi.org/10.1109/MCG.2009.120	journal	November 2009
In Situ Visualization for Large-Scale Combustion Simulations No authors listed IEEE Computer Graphics and Applications, Vol. 30, Issue 3 https://doi.org/10.1109/MCG.2010.55	journal	May 2010
Provenance for Visualizations: Reproducibility and Beyond Silva, Claudio T.; Freire, Juliana; Callahan, Steven P. Computing in Science & Engineering, Vol. 9, Issue 5 https://doi.org/10.1109/MCSE.2007.106	journal	September 2007
Grid Computing Workloads Iosup, Alexandru; Epema, Dick IEEE Internet Computing, Vol. 15, Issue 2 https://doi.org/10.1109/MIC.2010.130	journal	March 2011
The Hadoop Distributed File System Shvachko, Konstantin; Kuang, Hairong; Radia, Sanjay 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2010.5496972	conference	May 2010
Concurrent Visualization in a Production Supercomputing Environment Ellsworth, D.; Green, B.; Henze, C. IEEE Transactions on Visualization and Computer Graphics, Vol. 12, Issue 5 https://doi.org/10.1109/TVCG.2006.128	journal	September 2006
Characterization of scientific workflows Bharathi, Shishir; Chervenak, Ann; Deelman, Ewa 2008 Third Workshop on Workflows in Support of Large-Scale Science (WORKS 2008) https://doi.org/10.1109/WORKS.2008.4723958	conference	November 2008
dispel4py: An Agile Framework for Data-Intensive eScience Filgueira, Rosa; Krause, Amrey; Atkinson, Malcolm 2015 IEEE 11th International Conference on e-Science (e-Science) https://doi.org/10.1109/eScience.2015.40	conference	August 2015
Online Task Resource Consumption Prediction for Scientific Workflows da Silva, Rafael Ferreira; Juve, Gideon; Rynge, Mats Parallel Processing Letters, Vol. 25, Issue 03 https://doi.org/10.1142/S0129626415410030	journal	September 2015
Scientific workload characterization by loop-based analyses No authors listed ACM SIGMETRICS Performance Evaluation Review, Vol. 19, Issue 3 https://doi.org/10.1145/130951.130952	journal	February 1992
MapReduce: simplified data processing on large clusters Dean, Jeffrey; Ghemawat, Sanjay; Mehta, Brijesh Communications of the ACM, Vol. 51, Issue 1 https://doi.org/10.1145/1327452.1327492	journal	January 2008
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) Lofstead, Jay F.; Klasky, Scott; Schwan, Karsten Proceedings of the 6th international workshop on Challenges of large applications in distributed environments - CLADE '08 https://doi.org/10.1145/1383529.1383533	conference	January 2008
DataSpaces: an interaction and coordination framework for coupled simulation workflows Docan, Ciprian; Parashar, Manish; Klasky, Scott Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10 https://doi.org/10.1145/1851476.1851481	conference	January 2010
ParaTrac: a fine-grained profiler for data-intensive workflows Dun, Nan; Taura, Kenjiro; Yonezawa, Akinori Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10 https://doi.org/10.1145/1851476.1851482	conference	January 2010
ISABELA-QA: query-driven analytics with ISABELA-compressed extreme-scale scientific data Lakshminarasimhan, Sriram; Klasky, Scott; Latham, Robert Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063425	conference	January 2011
Apache airavata: a framework for distributed applications and computational workflows Marru, Suresh; Gardler, Ross; Slominski, Aleksander Proceedings of the 2011 ACM workshop on Gateway computing environments - GCE '11 https://doi.org/10.1145/2110486.2110490	conference	January 2011
Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids Albrecht, Michael; Donnelly, Patrick; Bui, Peter Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies - SWEET '12 https://doi.org/10.1145/2443416.2443417	conference	January 2012
Bobolang: a language for parallel streaming applications Falt, Zbyněk; Bednárek, David; Kruliš, Martin Proceedings of the 23rd international symposium on High-performance parallel and distributed computing - HPDC '14 https://doi.org/10.1145/2600212.2600711	conference	January 2014
Exascale computing and big data Reed, Daniel A.; Dongarra, Jack Communications of the ACM, Vol. 58, Issue 7 https://doi.org/10.1145/2699414	journal	June 2015
Towards Reproducibility in Scientific Workflows: An Infrastructure-Based Approach Santana-Perez, Idafen; Pérez-Hernández, María S. Scientific Programming, Vol. 2015 https://doi.org/10.1155/2015/243180	journal	January 2015
A Pipeline Virtual Service Pre-Scheduling Pattern and its Application in Astronomy Data Processing Wang, Man; Du, Zhihui; Cheng, Zhili SIMULATION, Vol. 83, Issue 1, p. 123-132 https://doi.org/10.1177/0037549707079234	journal	January 2007
Flexible and Efficient Workflow Deployment of Data-Intensive Applications On Grids With MOTEUR Glatard, Tristan; Montagnat, Johan; Lingrand, Diane The International Journal of High Performance Computing Applications, Vol. 22, Issue 3 https://doi.org/10.1177/1094342008096067	journal	August 2008
PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows Deelman, Ewa; Carothers, Christopher; Mandal, Anirban The International Journal of High Performance Computing Applications, Vol. 31, Issue 1 https://doi.org/10.1177/1094342015594515	journal	July 2016
Uncertainty Quantification in Computational Predictive Models for Fluid Dynamics Using a Workflow Management Engine Guerra, Gabriel; Rochinha, Fernando A.; Elias, Renato International Journal for Uncertainty Quantification, Vol. 2, Issue 1 https://doi.org/10.1615/Int.J.UncertaintyQuantification.v2.i1.50	journal	January 2012
Developing a Learning Algorithm-Generated Empirical Relaxer Mitchell, Wayne; Kallman, Josh; Toreja, Allen https://doi.org/10.2172/1248278	report	March 2016

Cited By (12)

Workflow provenance in the lifecycle of scientific machine learning Souza, Renan; Azevedo, Leonardo G.; Lourenço, Vítor Concurrency and Computation: Practice and Experience, Vol. 34, Issue 14 https://doi.org/10.1002/cpe.6544	journal	August 2021
Scientific workflows: Past, present and future Atkinson, Malcolm; Gesing, Sandra; Montagnat, Johan Future Generation Computer Systems, Vol. 75 https://doi.org/10.1016/j.future.2017.05.041	journal	October 2017
Systematically linking tranSMART, Galaxy and EGA for reusing human translational research data Zhang, Chao; Bijlard, Jochem; Staiger, Christine F1000Research, Vol. 6 https://doi.org/10.12688/f1000research.12168.1	journal	January 2017
HPC Application Cloudification: The StreamFlow Toolkit (Invited Paper) Colonnelli, Iacopo; Cantalupo, Barbara; Esposito, Roberto Schloss Dagstuhl - Leibniz-Zentrum für Informatik https://doi.org/10.4230/oasics.parma-ditam.2021.5	text	January 2021
Bringing AI pipelines onto cloud-HPC: setting a baseline for accuracy of COVID-19 diagnosis Colonnelli, Iacopo; Cantalupo, Barbara; Spampinato, Concetto Zenodo https://doi.org/10.5281/zenodo.5151511	text	January 2021
Ranking open source application integration frameworks based on maintainability metrics: A review of five‐year evolution Frantz, Rafael Z.; Rehbein, Matheus H.; Berlezi, Rodolfo Software: Practice and Experience, Vol. 49, Issue 10 https://doi.org/10.1002/spe.2733	journal	July 2019
Highly Interactive, Steered Scientific Workflows on HPC Systems: Optimizing Design Solutions Ossyra, John R.; Sedova, Ada; Baker, Matthew B. High Performance Computing: ISC High Performance 2019 International Workshops, Frankfurt, Germany, June 16-20, 2019, Revised Selected Papers, p. 514-527 https://doi.org/10.1007/978-3-030-34356-9_39	book	December 2019
BIGGR: Bringing Gradoop to Applications Rostami, M. Ali; Kricke, Matthias; Peukert, Eric Datenbank-Spektrum, Vol. 19, Issue 1 https://doi.org/10.1007/s13222-019-00306-x	journal	February 2019
Managing genomic variant calling workflows with Swift/T Ahmed, Azza; Heldenbrand, Jacob; Asmann, Yan PLOS ONE https://doi.org/10.1101/524645	journal	January 2019
Exploiting Spark for HPC Simulation Data: Taming the Ephemeral Data Explosion Jiang, Ming; Gallagher, Brian; Chu, Albert HPCAsia2020: International Conference on High Performance Computing in Asia-Pacific Region, Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region https://doi.org/10.1145/3368474.3368482	conference	January 2020
Managing genomic variant calling workflows with Swift/T Ahmed, Azza E.; Heldenbrand, Jacob; Asmann, Yan PLOS ONE, Vol. 14, Issue 7 https://doi.org/10.1371/journal.pone.0211608	journal	July 2019
Scientific workflows applied to the coupling of a continuum (Elmer v8.3) and a discrete element (HiDEM v1.0) ice dynamic model Memon, Shahbaz; Vallot, Dorothée; Zwinger, Thomas Geoscientific Model Development, Vol. 12, Issue 7 https://doi.org/10.5194/gmd-12-3001-2019	journal	January 2019

Similar Records

Running Ensemble Workflows at Extreme Scale: Lessons Learned and Path Forward

Conference · Sat Oct 01 00:00:00 EDT 2022 · OSTI ID:1968703

Extreme-scale workflows: A perspective from the JLESC international community

Journal Article · Mon Jul 22 20:00:00 EDT 2024 · Future Generations Computer Systems · OSTI ID:2440426

PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows

Journal Article · Mon Jul 13 20:00:00 EDT 2015 · International Journal of High Performance Computing Applications · OSTI ID:1265426

Related Subjects

97 MATHEMATICS AND COMPUTING
Extreme-scale computing
Scientific workflows
Workflow management systems
in situ processing

A characterization of workflow management systems for extreme-scale applications

Citation Formats

References (51)

Cited By (12)

Similar Records

Related Subjects