Pegasus, a workflow management system for science automation

Deelman, Ewa; Vahi, Karan; Juve, Gideon; Rynge, Mats; Callaghan, Scott; Maechling, Philip J.; Mayani, Rajiv; Chen, Weiwei; Ferreira da Silva, Rafael; Livny, Miron; Wenger, Kent

doi:10.1016/j.future.2014.10.008

Title: Pegasus, a workflow management system for science automation

Journal Article · Fri May 01 00:00:00 EDT 2015 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2014.10.008· OSTI ID:1245675

Deelman, Ewa; Vahi, Karan; Juve, Gideon; Rynge, Mats; Callaghan, Scott; Maechling, Philip J.; Mayani, Rajiv; Chen, Weiwei;

;

; Wenger, Kent

View Accepted Manuscript (Publisher)

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 1245675

Journal Information:: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Vol. 46 Journal Issue: C; ISSN 0167-739X

Publisher:: ElsevierCopyright Statement

Country of Publication:: Netherlands

Language:: English

Citation Metrics:

Cited by: 432 works

Citation information provided by
Web of Science

References (46)

The Application of Cloud Computing to Astronomy: A Study of Cost and Performance Berriman, G. Bruce; Juve, Gideon; Deelman, Ewa 2010 Sixth IEEE International Conference on E-Science Workshops https://doi.org/10.1109/eScienceW.2010.10	conference	December 2010
ASKALON: a Grid application development and computing environment Fahringer, T.; Prodan, R. The 6th IEEE/ACM International Workshop on Grid Computing, 2005. https://doi.org/10.1109/GRID.2005.1542733	conference	January 2005
Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids Albrecht, Michael; Donnelly, Patrick; Bui, Peter Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies - SWEET '12 https://doi.org/10.1145/2443416.2443417	conference	January 2012
On the granularity and clustering of directed acyclic task graphs Gerasoulis, A.; Yang, T. IEEE Transactions on Parallel and Distributed Systems, Vol. 4, Issue 6 https://doi.org/10.1109/71.242154	journal	June 1993
Parameter Exploration in Science and Engineering Using Many-Task Computing Abramson, D.; Bethwaite, B.; Enticott, C. IEEE Transactions on Parallel and Distributed Systems, Vol. 22, Issue 6 https://doi.org/10.1109/TPDS.2010.177	journal	June 2011
Query processing in a system for distributed databases (SDD-1) Bernstein, Philip A.; Goodman, Nathan; Wong, Eugene ACM Transactions on Database Systems, Vol. 6, Issue 4 https://doi.org/10.1145/319628.319650	journal	December 1981
Enabling large-scale scientific workflows on petascale resources using MPI master/worker Rynge, Mats; Callaghan, Scott; Deelman, Ewa Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment on Bridging from the eXtreme to the campus and beyond - XSEDE '12 https://doi.org/10.1145/2335755.2335846	conference	January 2012
Static scheduling algorithms for allocating directed task graphs to multiprocessors Kwok, Yu-Kwong; Ahmad, Ishfaq ACM Computing Surveys, Vol. 31, Issue 4 https://doi.org/10.1145/344588.344618	journal	December 1999
Tavaxy: Integrating Taverna and Galaxy workflows with cloud computing support Abouelhoda, Mohamed; Issa, Shadi Alaa; Ghanem, Moustafa BMC Bioinformatics, Vol. 13, Issue 1 https://doi.org/10.1186/1471-2105-13-77	journal	May 2012
A resource management architecture for metacomputing systems Czajkowski, Karl; Foster, Ian; Karonis, Nick Job Scheduling Strategies for Parallel Processing https://doi.org/10.1007/BFb0053981	book	January 1998
Experiences Using GlideinWMS and the Corral Frontend across Cyberinfrastructures Rynge, Mats; Juve, Gideon; Mehta, Gaurang 2011 IEEE 7th International Conference on E-Science (e-Science), 2011 IEEE Seventh International Conference on eScience https://doi.org/10.1109/eScience.2011.50	conference	December 2011
Wings: Intelligent Workflow-Based Design of Computational Experiments Gil, Yolanda; Ratnakar, Varun; Kim, Jihie IEEE Intelligent Systems, Vol. 26, Issue 1 https://doi.org/10.1109/MIS.2010.9	journal	January 2011
Wide area data replication for scientific collaborations Chervenak, A.; Schuler, R.; Kesselman, C. The 6th IEEE/ACM International Workshop on Grid Computing, 2005. https://doi.org/10.1109/GRID.2005.1542717	conference	January 2005
High-Throughput, Kingdom-Wide Prediction and Annotation of Bacterial Non-Coding RNAs Livny, Jonathan; Teonadi, Hidayat; Livny, Miron PLoS ONE, Vol. 3, Issue 9 https://doi.org/10.1371/journal.pone.0003197	journal	September 2008
Accessing opportunistic resources with Bosco Weitzel, D.; Sfiligoi, I.; Bockelman, B. Journal of Physics: Conference Series, Vol. 513, Issue 3 https://doi.org/10.1088/1742-6596/513/3/032105	journal	June 2014
Scaling up workflow-based applications Callaghan, Scott; Deelman, Ewa; Gunter, Dan Journal of Computer and System Sciences, Vol. 76, Issue 6 https://doi.org/10.1016/j.jcss.2009.11.005	journal	September 2010
A view of cloud computing Armbrust, Michael; Stoica, Ion; Zaharia, Matei Communications of the ACM, Vol. 53, Issue 4 https://doi.org/10.1145/1721654.1721672	journal	April 2010
Performance-effective and low-complexity task scheduling for heterogeneous computing Topcuoglu, H.; Hariri, S. IEEE Transactions on Parallel and Distributed Systems, Vol. 13, Issue 3, p. 260-274 https://doi.org/10.1109/71.993206	journal	March 2002
Provenance: The Bridge Between Experiments and Data Miles, Simon; Groth, Paul; Deelman, Ewa Computing in Science & Engineering, Vol. 10, Issue 3 https://doi.org/10.1109/MCSE.2008.82	journal	May 2008
Characterizing and profiling scientific workflows Juve, Gideon; Chervenak, Ann; Deelman, Ewa Future Generation Computer Systems, Vol. 29, Issue 3 https://doi.org/10.1016/j.future.2012.08.015	journal	March 2013
Workflows and e-Science: An overview of workflow system features and capabilities Deelman, Ewa; Gannon, Dennis; Shields, Matthew Future Generation Computer Systems, Vol. 25, Issue 5 https://doi.org/10.1016/j.future.2008.06.012	journal	May 2009
Workflow task clustering for best effort systems with Pegasus Singh, Gurmeet; Su, Mei-Hui; Vahi, Karan Proceedings of the 15th ACM Mardi Gras conference on From lightweight mash-ups to lambda grids: Understanding the spectrum of distributed computing requirements, applications, tools, infrastructures, interoperability, and the incremental adoption of key capabilities - MG '08 https://doi.org/10.1145/1341811.1341822	conference	January 2008
The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud Wolstencroft, Katherine; Haines, Robert; Fellows, Donal Nucleic Acids Research, Vol. 41, Issue W1 https://doi.org/10.1093/nar/gkt328	journal	May 2013
A Case Study into Using Common Real-Time Workflow Monitoring Infrastructure for Scientific Workflows Vahi, Karan; Harvey, Ian; Samak, Taghrid Journal of Grid Computing, Vol. 11, Issue 3 https://doi.org/10.1007/s10723-013-9265-4	journal	June 2013
The open science grid Pordes, Ruth; Petravick, Don; Kramer, Bill Journal of Physics: Conference Series, Vol. 78 https://doi.org/10.1088/1742-6596/78/1/012057	journal	July 2007
Hypertext Transfer Protocol -- HTTP/1.1 Fielding, R.; Gettys, J.; Mogul, J. https://doi.org/10.17487/rfc2616	report	June 1999
Workflow overhead analysis and optimizations Chen, Weiwei; Deelman, Ewa Proceedings of the 6th workshop on Workflows in support of large-scale science - WORKS '11 https://doi.org/10.1145/2110497.2110500	conference	January 2011
A Taxonomy of Workflow Management Systems for Grid Computing Yu, Jia; Buyya, Rajkumar Journal of Grid Computing, Vol. 3, Issue 3-4 https://doi.org/10.1007/s10723-005-9010-8	journal	September 2005
Scheduling Data-IntensiveWorkflows onto Storage-Constrained Distributed Resources Ramakrishnan, Arun; Singh, Gurmeet; Zhao, Henan Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07) https://doi.org/10.1109/CCGRID.2007.101	conference	May 2007
Rethinking data management for big data scientific workflows Vahi, Karan; Rynge, Mats; Juve, Gideon 2013 IEEE International Conference on Big Data https://doi.org/10.1109/BigData.2013.6691724	conference	October 2013
CyberShake: A Physics-Based Seismic Hazard Model for Southern California Graves, Robert; Jordan, Thomas H.; Callaghan, Scott Pure and Applied Geophysics, Vol. 168, Issue 3-4 https://doi.org/10.1007/s00024-010-0161-6	journal	May 2010
A Scalable Approach to Network Enabled Servers Caron, Eddy; Desprez, Frédéric; Lombard, Frédéric Euro-Par 2002 Parallel Processing https://doi.org/10.1007/3-540-45706-2_128	book	January 2002
Balanced Task Clustering in Scientific Workflows Chen, Weiwei; Silva, Rafael Ferreira Da; Deelman, Ewa 2013 IEEE 9th International Conference on eScience (eScience), 2013 IEEE 9th International Conference on e-Science https://doi.org/10.1109/eScience.2013.40	conference	October 2013
glideinWMS—a generic pilot-based workload management system Sfiligoi, I. Journal of Physics: Conference Series, Vol. 119, Issue 6 https://doi.org/10.1088/1742-6596/119/6/062044	journal	July 2008
Apache airavata: a framework for distributed applications and computational workflows Marru, Suresh; Gardler, Ross; Slominski, Aleksander Proceedings of the 2011 ACM workshop on Gateway computing environments - GCE '11 https://doi.org/10.1145/2110486.2110490	conference	January 2011
Flexible and Efficient Workflow Deployment of Data-Intensive Applications On Grids With MOTEUR Glatard, Tristan; Montagnat, Johan; Lingrand, Diane The International Journal of High Performance Computing Applications, Vol. 22, Issue 3 https://doi.org/10.1177/1094342008096067	journal	August 2008
The Secure Shell (SSH) Connection Protocol Ylonen, T.; Lonvick, C. https://doi.org/10.17487/rfc4254	report	January 2006
Toward fine-grained online task characteristics estimation in scientific workflows da Silva, Rafael Ferreira; Juve, Gideon; Deelman, Ewa Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science - WORKS '13 https://doi.org/10.1145/2534248.2534254	conference	January 2013
Distributed computing in practice: the Condor experience Thain, Douglas; Tannenbaum, Todd; Livny, Miron Concurrency and Computation: Practice and Experience, Vol. 17, Issue 2-4, p. 323-356 https://doi.org/10.1002/cpe.938	journal	January 2005
Connecting Scientific Data to Scientific Experiments with Provenance Miles, Simon; Deelman, Ewa; Groth, Paul Third IEEE International Conference on e-Science and Grid Computing (e-Science 2007) https://doi.org/10.1109/E-SCIENCE.2007.22	conference	December 2007
The Open Provenance Model core specification (v1.1) Moreau, Luc; Clifford, Ben; Freire, Juliana Future Generation Computer Systems, Vol. 27, Issue 6 https://doi.org/10.1016/j.future.2010.07.005	journal	June 2011
Dependence graphs and compiler optimizations Kuck, D. J.; Kuhn, R. H.; Padua, D. A. Proceedings of the 8th ACM SIGPLAN-SIGACT symposium on Principles of programming languages - POPL '81 https://doi.org/10.1145/567532.567555	conference	January 1981
The Anatomy of the Grid: Enabling Scalable Virtual Organizations Foster, Ian; Kesselman, Carl; Tuecke, Steven The International Journal of High Performance Computing Applications, Vol. 15, Issue 3 https://doi.org/10.1177/109434200101500302	journal	August 2001
Using imbalance metrics to optimize task clustering in scientific workflow executions Chen, Weiwei; Ferreira da Silva, Rafael; Deelman, Ewa Future Generation Computer Systems, Vol. 46 https://doi.org/10.1016/j.future.2014.09.014	journal	May 2015
Searching for gravitational waves from binary coalescence Babak, S.; Biswas, R.; Brady, P. R. Physical Review D, Vol. 87, Issue 2 https://doi.org/10.1103/PhysRevD.87.024033	journal	January 2013
Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences Goecks, Jeremy; Nekrutenko, Anton; Taylor, James Genome Biology, Vol. 11, Issue 8 https://doi.org/10.1186/gb-2010-11-8-r86	journal	January 2010

Similar Records

The Evolution of the Pegasus Workflow Management Software

Journal Article · Wed May 29 00:00:00 EDT 2019 · Computing in Science and Engineering · OSTI ID:1245675

Deelman, Ewa; Vahi, Karan; Rynge, Mats; +4 more

Progress and prospects for accelerating materials science with automated and autonomous workflows

Journal Article · Wed Oct 30 00:00:00 EDT 2019 · Chemical Science · OSTI ID:1245675

Stein, Helge S.; Gregoire, John M.

Building a Cyberinfrastructure for Environmental System Science: Modeling Frameworks, Data Management, and Scientific Workflows, Workshop Report, Potomac, Maryland, April 30-May 1, 2015

Technical Report · Mon Nov 30 00:00:00 EST 2015 · OSTI ID:1245675

Moulton, David; Williams, Dean; Agarwal, Deb; +8 more

Title: Pegasus, a workflow management system for science automation

Citation Formats

References (46)

Similar Records

Related Subjects