A characterization of workflow management systems for extreme-scale applications

Ferreira da Silva, Rafael; Filgueira, Rosa; Pietri, Ilia; Jiang, Ming; Sakellariou, Rizos; Deelman, Ewa

doi:10.1016/j.future.2017.02.026

Title: A characterization of workflow management systems for extreme-scale applications

Journal Article · 16 February 2017 · Future Generations Computer Systems

DOI: https://doi.org/10.1016/j.future.2017.02.026 · OSTI ID:1408072

Ferreira da Silva, Rafael ^[1]; Filgueira, Rosa ^[2]; Pietri, Ilia ^[3]; Jiang, Ming ^[4]; Sakellariou, Rizos ^[5]; Deelman, Ewa ^[1]

University of Southern California, Marina del Rey, CA (United States)
British Geological Survey, Lyell Centre, Edinburgh (United Kingdom); University of Edinburgh (United Kingdom). School of Informatics
Univ. of Athens (Greece). Department of Informatics and Telecommunication
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Univ. of Manchester (United Kingdom). School of Computer Science

We present that the automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today’s computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. Finally, the paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC52-07NA27344; SC0012636; AC52-07NA27344 (LLNL-JRNL-706700); 16-ERD-036; DESC0012636

OSTI ID:: 1408072

Alternate ID(s):: OSTI ID: 1495625

Report Number(s):: LLNL-JRNL-706700

Journal Information:: Future Generations Computer Systems, Vol. 75, Issue C; ISSN 0167-739X

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (51)

MapReduce: simplified data processing on large clusters Dean, Jeffrey; Ghemawat, Sanjay; Mehta, Brijesh Communications of the ACM, Vol. 51, Issue 1 https://doi.org/10.1145/1327452.1327492	journal	January 2008
Self-scalable services in service oriented software for cost-effective data farming Król, Dariusz; Kitowski, Jacek Future Generation Computer Systems, Vol. 54 https://doi.org/10.1016/j.future.2015.07.003	journal	January 2016
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) Lofstead, Jay F.; Klasky, Scott; Schwan, Karsten Proceedings of the 6th international workshop on Challenges of large applications in distributed environments - CLADE '08 https://doi.org/10.1145/1383529.1383533	conference	January 2008
A Supervised Learning Framework for Arbitrary Lagrangian-Eulerian Simulations Jiang, Ming; Gallagher, Brian; Kallman, Joshua 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA) https://doi.org/10.1109/ICMLA.2016.0176	conference	December 2016
FireWorks: a dynamic workflow system designed for high-throughput applications: FireWorks: A Dynamic Workflow System Designed for High-Throughput Applications Jain, Anubhav; Ong, Shyue Ping; Chen, Wei Concurrency and Computation: Practice and Experience, Vol. 27, Issue 17 https://doi.org/10.1002/cpe.3505	journal	May 2015
Taverna: a tool for the composition and enactment of bioinformatics workflows Oinn, T.; Addis, M.; Ferris, J. Bioinformatics, Vol. 20, Issue 17 https://doi.org/10.1093/bioinformatics/bth361	journal	June 2004
A Steering Environment for Online Parallel Visualization of Legacy Parallel Simulations Esnard, Aurelien; Richart, Nicolas; Coulaud, Olivier Proceedings. Tenth IEEE International Symposium on Distributed Simulation and Real-Time Applications, 2006 Tenth IEEE International Symposium on Distributed Simulation and Real-Time Applications https://doi.org/10.1109/DS-RT.2006.7	conference	October 2006
Swift: A language for distributed parallel scripting Wilde, Michael; Hategan, Mihael; Wozniak, Justin M. Parallel Computing, Vol. 37, Issue 9 https://doi.org/10.1016/j.parco.2011.05.005	journal	September 2011
PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows Deelman, Ewa; Carothers, Christopher; Mandal, Anirban The International Journal of High Performance Computing Applications, Vol. 31, Issue 1 https://doi.org/10.1177/1094342015594515	journal	July 2016
DataSpaces: an interaction and coordination framework for coupled simulation workflows Docan, Ciprian; Parashar, Manish; Klasky, Scott Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10 https://doi.org/10.1145/1851476.1851481	conference	January 2010
ParaTrac: a fine-grained profiler for data-intensive workflows Dun, Nan; Taura, Kenjiro; Yonezawa, Akinori Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10 https://doi.org/10.1145/1851476.1851482	conference	January 2010
A Survey of Data-Intensive Scientific Workflow Management Liu, Ji; Pacitti, Esther; Valduriez, Patrick Journal of Grid Computing, Vol. 13, Issue 4 https://doi.org/10.1007/s10723-015-9329-8	journal	March 2015
Grid Computing Workloads Iosup, Alexandru; Epema, Dick IEEE Internet Computing, Vol. 15, Issue 2 https://doi.org/10.1109/MIC.2010.130	journal	March 2011
Characterizing and profiling scientific workflows Juve, Gideon; Chervenak, Ann; Deelman, Ewa Future Generation Computer Systems, Vol. 29, Issue 3 https://doi.org/10.1016/j.future.2012.08.015	journal	March 2013
Workflows and e-Science: An overview of workflow system features and capabilities Deelman, Ewa; Gannon, Dennis; Shields, Matthew Future Generation Computer Systems, Vol. 25, Issue 5 https://doi.org/10.1016/j.future.2008.06.012	journal	May 2009
Science automation in practice: Performance data farming in workflows Krol, Dariusz; Kitowski, Jacek; da Silva, Rafael Ferreira 2016 IEEE 21st International Conference on Emerging Technologies and Factory Automation (ETFA) https://doi.org/10.1109/ETFA.2016.7733677	conference	September 2016
The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud Wolstencroft, Katherine; Haines, Robert; Fellows, Donal Nucleic Acids Research, Vol. 41, Issue W1 https://doi.org/10.1093/nar/gkt328	journal	May 2013
Taverna: a tool for building and running workflows of services Hull, D.; Wolstencroft, K.; Stevens, R. Nucleic Acids Research, Vol. 34, Issue Web Server https://doi.org/10.1093/nar/gkl320	journal	July 2006
dispel4py: An Agile Framework for Data-Intensive eScience Filgueira, Rosa; Krause, Amrey; Atkinson, Malcolm 2015 IEEE 11th International Conference on e-Science (e-Science) https://doi.org/10.1109/eScience.2015.40	conference	August 2015
Characterization of scientific workflows Bharathi, Shishir; Chervenak, Ann; Deelman, Ewa 2008 Third Workshop on Workflows in Support of Large-Scale Science (WORKS 2008) https://doi.org/10.1109/WORKS.2008.4723958	conference	November 2008
Apache airavata: a framework for distributed applications and computational workflows Marru, Suresh; Gardler, Ross; Slominski, Aleksander Proceedings of the 2011 ACM workshop on Gateway computing environments - GCE '11 https://doi.org/10.1145/2110486.2110490	conference	January 2011
Bobolang: a language for parallel streaming applications Falt, Zbyněk; Bednárek, David; Kruliš, Martin Proceedings of the 23rd international symposium on High-performance parallel and distributed computing - HPDC '14 https://doi.org/10.1145/2600212.2600711	conference	January 2014
The ParaView Coprocessing Library: A scalable, general purpose in situ visualization library Fabian, Nathan; Moreland, Kenneth; Thompson, David 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092322	conference	October 2011
Uncertainty Quantification in Computational Predictive Models for Fluid Dynamics Using a Workflow Management Engine Guerra, Gabriel; Rochinha, Fernando A.; Elias, Renato International Journal for Uncertainty Quantification, Vol. 2, Issue 1 https://doi.org/10.1615/Int.J.UncertaintyQuantification.v2.i1.50	journal	January 2012
Adaptable, metadata rich IO methods for portable high performance IO Lofstead, Jay; Zheng, Fang; Klasky, Scott Distributed Processing (IPDPS), 2009 IEEE International Symposium on Parallel & Distributed Processing https://doi.org/10.1109/IPDPS.2009.5161052	conference	May 2009
The Hadoop Distributed File System Shvachko, Konstantin; Kuang, Hairong; Radia, Sanjay 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2010.5496972	conference	May 2010
Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids Albrecht, Michael; Donnelly, Patrick; Bui, Peter Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies - SWEET '12 https://doi.org/10.1145/2443416.2443417	conference	January 2012
Scientific workload characterization by loop-based analyses ACM SIGMETRICS Performance Evaluation Review, Vol. 19, Issue 3 https://doi.org/10.1145/130951.130952	journal	February 1992
The Spallation Neutron Source in Oak Ridge: A powerful tool for materials research Mason, T. E.; Abernathy, D.; Anderson, I. Physica B: Condensed Matter, Vol. 385-386 https://doi.org/10.1016/j.physb.2006.05.281	journal	November 2006
Exascale computing and big data Reed, Daniel A.; Dongarra, Jack Communications of the ACM, Vol. 58, Issue 7 https://doi.org/10.1145/2699414	journal	June 2015
Pegasus, a workflow management system for science automation Deelman, Ewa; Vahi, Karan; Juve, Gideon Future Generation Computer Systems, Vol. 46 https://doi.org/10.1016/j.future.2014.10.008	journal	May 2015
Galaxy: A Web‐Based Genome Analysis Tool for Experimentalists Blankenberg, Daniel; Kuster, Gregory Von; Coraor, Nathaniel Current Protocols in Molecular Biology, Vol. 89, Issue 1 https://doi.org/10.1002/0471142727.mb1910s89	journal	January 2010
Enabling In-situ Execution of Coupled Scientific Workflow on Multi-core Platform Zhang, Fan; Docan, Ciprian; Parashar, Manish 2012 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), 2012 IEEE 26th International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2012.122	conference	May 2012
Developing a Learning Algorithm-Generated Empirical Relaxer Mitchell, Wayne; Kallman, Josh; Toreja, Allen https://doi.org/10.2172/1248278	report	March 2016
Algorithms for cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds Malawski, Maciej; Juve, Gideon; Deelman, Ewa Future Generation Computer Systems, Vol. 48 https://doi.org/10.1016/j.future.2015.01.004	journal	July 2015
Practical Resource Monitoring for Robust High Throughput Computing Juve, Gideon; Tovar, Benjamin; Silva, Rafael Ferreira da 2015 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2015.115	conference	September 2015
Workflows and extensions to the Kepler scientific workflow system to support environmental sensor data access and analysis Barseghian, Derik; Altintas, Ilkay; Jones, Matthew B. Ecological Informatics, Vol. 5, Issue 1 https://doi.org/10.1016/j.ecoinf.2009.08.008	journal	January 2010
Toward simulation-time data analysis and I/O acceleration on leadership-class systems Vishwanath, Venkatram; Hereld, Mark; Papka, Michael E. 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092178	conference	October 2011
Using a suite of ontologies for preserving workflow-centric research objects Belhajjame, Khalid; Zhao, Jun; Garijo, Daniel Journal of Web Semantics, Vol. 32 https://doi.org/10.1016/j.websem.2015.01.003	journal	May 2015
ISABELA-QA: query-driven analytics with ISABELA-compressed extreme-scale scientific data Lakshminarasimhan, Sriram; Klasky, Scott; Latham, Robert Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063425	conference	January 2011
A Pipeline Virtual Service Pre-Scheduling Pattern and its Application in Astronomy Data Processing Wang, Man; Du, Zhihui; Cheng, Zhili SIMULATION, Vol. 83, Issue 1, p. 123-132 https://doi.org/10.1177/0037549707079234	journal	January 2007
A Taxonomy of Workflow Management Systems for Grid Computing Yu, Jia; Buyya, Rajkumar Journal of Grid Computing, Vol. 3, Issue 3-4 https://doi.org/10.1007/s10723-005-9010-8	journal	September 2005
In Situ Visualization at Extreme Scale: Challenges and Opportunities IEEE Computer Graphics and Applications, Vol. 29, Issue 6 https://doi.org/10.1109/MCG.2009.120	journal	November 2009
In Situ Visualization for Large-Scale Combustion Simulations IEEE Computer Graphics and Applications, Vol. 30, Issue 3 https://doi.org/10.1109/MCG.2010.55	journal	May 2010
SciCumulus: A Lightweight Cloud Middleware to Explore Many Task Computing Paradigm in Scientific Workflows de Oliveira, Daniel; Ogasawara, Eduardo; Baião, Fernanda 2010 IEEE International Conference on Cloud Computing (CLOUD), 2010 IEEE 3rd International Conference on Cloud Computing https://doi.org/10.1109/CLOUD.2010.64	conference	July 2010
Concurrent Visualization in a Production Supercomputing Environment Ellsworth, D.; Green, B.; Henze, C. IEEE Transactions on Visualization and Computer Graphics, Vol. 12, Issue 5 https://doi.org/10.1109/TVCG.2006.128	journal	September 2006
Provenance for Visualizations: Reproducibility and Beyond Silva, Claudio T.; Freire, Juliana; Callahan, Steven P. Computing in Science & Engineering, Vol. 9, Issue 5 https://doi.org/10.1109/MCSE.2007.106	journal	September 2007
Flexible and Efficient Workflow Deployment of Data-Intensive Applications On Grids With MOTEUR Glatard, Tristan; Montagnat, Johan; Lingrand, Diane The International Journal of High Performance Computing Applications, Vol. 22, Issue 3 https://doi.org/10.1177/1094342008096067	journal	August 2008
Towards Reproducibility in Scientific Workflows: An Infrastructure-Based Approach Santana-Perez, Idafen; Pérez-Hernández, María S. Scientific Programming, Vol. 2015 https://doi.org/10.1155/2015/243180	journal	January 2015
Online Task Resource Consumption Prediction for Scientific Workflows da Silva, Rafael Ferreira; Juve, Gideon; Rynge, Mats Parallel Processing Letters, Vol. 25, Issue 03 https://doi.org/10.1142/S0129626415410030	journal	September 2015
Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-Intensive Science Filgueira, Rosa; Da Silva, Rafael Ferreira; Krause, Amrey 2016 Seventh International Workshop on Data-Intensive Computing in the Clouds (DataCloud) https://doi.org/10.1109/DataCloud.2016.004	conference	November 2016

Cited By (13)

HPC Application Cloudification: The StreamFlow Toolkit (Invited Paper) Colonnelli, Iacopo; Cantalupo, Barbara; Esposito, Roberto Schloss Dagstuhl - Leibniz-Zentrum für Informatik https://doi.org/10.4230/oasics.parma-ditam.2021.5	text	January 2021
Bringing AI pipelines onto cloud-HPC: setting a baseline for accuracy of COVID-19 diagnosis Colonnelli, Iacopo; Cantalupo, Barbara; Spampinato, Concetto Zenodo https://doi.org/10.5281/zenodo.5151511	text	January 2021
Workflow provenance in the lifecycle of scientific machine learning Souza, Renan; Azevedo, Leonardo G.; Lourenço, Vítor Concurrency and Computation: Practice and Experience, Vol. 34, Issue 14 https://doi.org/10.1002/cpe.6544	journal	August 2021
Managing genomic variant calling workflows with Swift/T Ahmed, Azza E.; Heldenbrand, Jacob; Asmann, Yan PLOS ONE, Vol. 14, Issue 7 https://doi.org/10.1371/journal.pone.0211608	journal	July 2019
Scientific workflows: Past, present and future Atkinson, Malcolm; Gesing, Sandra; Montagnat, Johan Future Generation Computer Systems, Vol. 75 https://doi.org/10.1016/j.future.2017.05.041	journal	October 2017
Exploiting Spark for HPC Simulation Data: Taming the Ephemeral Data Explosion Jiang, Ming; Gallagher, Brian; Chu, Albert HPCAsia2020: International Conference on High Performance Computing in Asia-Pacific Region, Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region https://doi.org/10.1145/3368474.3368482	conference	January 2020
Managing genomic variant calling workflows with Swift/T Ahmed, Azza; Heldenbrand, Jacob; Asmann, Yan PLOS ONE https://doi.org/10.1101/524645	journal	January 2019
Highly Interactive, Steered Scientific Workflows on HPC Systems: Optimizing Design Solutions Ossyra, John R.; Sedova, Ada; Baker, Matthew B. High Performance Computing: ISC High Performance 2019 International Workshops, Frankfurt, Germany, June 16-20, 2019, Revised Selected Papers, p. 514-527 https://doi.org/10.1007/978-3-030-34356-9_39	book	December 2019
Systematically linking tranSMART, Galaxy and EGA for reusing human translational research data Zhang, Chao; Bijlard, Jochem; Staiger, Christine F1000Research, Vol. 6 https://doi.org/10.12688/f1000research.12168.1	journal	January 2017
Ranking open source application integration frameworks based on maintainability metrics: A review of five‐year evolution Frantz, Rafael Z.; Rehbein, Matheus H.; Berlezi, Rodolfo Software: Practice and Experience, Vol. 49, Issue 10 https://doi.org/10.1002/spe.2733	journal	July 2019
BIGGR: Bringing Gradoop to Applications Rostami, M. Ali; Kricke, Matthias; Peukert, Eric Datenbank-Spektrum, Vol. 19, Issue 1 https://doi.org/10.1007/s13222-019-00306-x	journal	February 2019
Scientific workflows applied to the coupling of a continuum (Elmer v8.3) and a discrete element (HiDEM v1.0) ice dynamic model Memon, Shahbaz; Vallot, Dorothée; Zwinger, Thomas Geoscientific Model Development, Vol. 12, Issue 7 https://doi.org/10.5194/gmd-12-3001-2019	journal	January 2019
Systematically linking tranSMART, Galaxy and EGA for reusing human translational research data Zhang, Chao; Bijlard, Jochem; Staiger, Christine F1000Research, Vol. 6 https://doi.org/10.12688/f1000research.12168.1	journal	January 2017

Similar Records

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows

Journal Article · 2019 · Future Generations Computer Systems · OSTI ID:1611913

Ferreira da Silva, Rafael; Filgueira, Rosa; Deelman, Ewa; +3 more

PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows

Journal Article · 2015 · International Journal of High Performance Computing Applications · OSTI ID:1265426

Deelman, Ewa; Carothers, Christopher; Mandal, Anirban; +12 more

Running Ensemble Workflows at Extreme Scale: Lessons Learned and Path Forward

Conference · 2022 · OSTI ID:1968703

Mehta, Kshitij; Cliff, Ashley; Suter, Fred; +4 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Scientific workflows
Workflow management systems
Extreme-scale computing
in situ processing

Title: A characterization of workflow management systems for extreme-scale applications

Citation Formats

References (51)

Cited By (13)

Similar Records

Related Subjects