skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Working with Workflows: Highlights from 5 years Building Scientific Workflows

Abstract

In 2006, the SciDAC Scientific Data Management (SDM) Center proposed to continue its work deploying leading edge data management and analysis capabilities to scientific applications. One of three thrust areas within the proposed center was focused on Scientific Process Automation (SPA) using workflow technology. As a founding member of the Kepler consortium [LAB+09], the SDM Center team was well positioned to begin deploying workflows immediately. We were also keenly aware of some of the deficiencies in Kepler when applied to high performance computing workflows, which allowed us to focus our research and development efforts on critical new capabilities which were ultimately integrated into the Kepler open source distribution, benefiting the entire community. Significant work was required to ensure Kepler was capable of supporting large-scale production runs for SciDAC applications. Our work on generic actors and templates have improved the portability of workflows across machines and provided a higher level of abstraction for workflow developers. Fault tolerance and provenance tracking were obvious areas for improvement within Kepler given the longevity and complexity of our target workflows. To monitor workflow execution, we developed and deployed a web-based dashboard. We then generalized this interface and released it so it could be deployed atmore » other locations. Outreach has always been a primary focus of our work and we had many successful deployments across a number of scientific domains while continually publishing and presenting our work. This short paper describes our most significant accomplishments over the past 5 years. Additional information about the SDM Center can be found in the companion paper: The Scientific Data Management Center: Available Technologies and Highlights.« less

Authors:
; ; ; ; ; ; ; ; ; ; ; ; ; ; ;
Publication Date:
Research Org.:
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1036427
Report Number(s):
PNNL-SA-80673
KJ0403000; TRN: US201206%%300
DOE Contract Number:  
AC05-76RL01830
Resource Type:
Conference
Resource Relation:
Conference: Proceedings of the 2011 SciDAC Conference, July 10-14, 2011, Denver, Colorado
Country of Publication:
United States
Language:
English
Subject:
99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; AUTOMATION; DATA BASE MANAGEMENT; DISTRIBUTION; MANAGEMENT; MONITORS; PERFORMANCE; PRODUCTION; TARGETS; TOLERANCE

Citation Formats

Critchlow, Terence J., Altintas, Ilkay, Chin, George, Crawl, Daniel, Iyer, H., Khan, Ayla, Klasky, S., Koehler, Sven, Ludaescher, Bertram T., Mouallem, Pierre, Nagappan, Mie, Podhorszki, Norbert, Shoshani, Arie, Silva, C., Tchoua, Roselynne, and Vouk, M. Working with Workflows: Highlights from 5 years Building Scientific Workflows. United States: N. p., 2011. Web.
Critchlow, Terence J., Altintas, Ilkay, Chin, George, Crawl, Daniel, Iyer, H., Khan, Ayla, Klasky, S., Koehler, Sven, Ludaescher, Bertram T., Mouallem, Pierre, Nagappan, Mie, Podhorszki, Norbert, Shoshani, Arie, Silva, C., Tchoua, Roselynne, & Vouk, M. Working with Workflows: Highlights from 5 years Building Scientific Workflows. United States.
Critchlow, Terence J., Altintas, Ilkay, Chin, George, Crawl, Daniel, Iyer, H., Khan, Ayla, Klasky, S., Koehler, Sven, Ludaescher, Bertram T., Mouallem, Pierre, Nagappan, Mie, Podhorszki, Norbert, Shoshani, Arie, Silva, C., Tchoua, Roselynne, and Vouk, M. Sat . "Working with Workflows: Highlights from 5 years Building Scientific Workflows". United States.
@article{osti_1036427,
title = {Working with Workflows: Highlights from 5 years Building Scientific Workflows},
author = {Critchlow, Terence J. and Altintas, Ilkay and Chin, George and Crawl, Daniel and Iyer, H. and Khan, Ayla and Klasky, S. and Koehler, Sven and Ludaescher, Bertram T. and Mouallem, Pierre and Nagappan, Mie and Podhorszki, Norbert and Shoshani, Arie and Silva, C. and Tchoua, Roselynne and Vouk, M.},
abstractNote = {In 2006, the SciDAC Scientific Data Management (SDM) Center proposed to continue its work deploying leading edge data management and analysis capabilities to scientific applications. One of three thrust areas within the proposed center was focused on Scientific Process Automation (SPA) using workflow technology. As a founding member of the Kepler consortium [LAB+09], the SDM Center team was well positioned to begin deploying workflows immediately. We were also keenly aware of some of the deficiencies in Kepler when applied to high performance computing workflows, which allowed us to focus our research and development efforts on critical new capabilities which were ultimately integrated into the Kepler open source distribution, benefiting the entire community. Significant work was required to ensure Kepler was capable of supporting large-scale production runs for SciDAC applications. Our work on generic actors and templates have improved the portability of workflows across machines and provided a higher level of abstraction for workflow developers. Fault tolerance and provenance tracking were obvious areas for improvement within Kepler given the longevity and complexity of our target workflows. To monitor workflow execution, we developed and deployed a web-based dashboard. We then generalized this interface and released it so it could be deployed at other locations. Outreach has always been a primary focus of our work and we had many successful deployments across a number of scientific domains while continually publishing and presenting our work. This short paper describes our most significant accomplishments over the past 5 years. Additional information about the SDM Center can be found in the companion paper: The Scientific Data Management Center: Available Technologies and Highlights.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2011},
month = {7}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: