Services + Components = Data Intensive Scientific Workflow Applications with MeDICi

Conference · Mon Jun 01 04:00:00 EDT 2009

DOI:https://doi.org/10.1007/978-3-642-02414-6_14· OSTI ID:1011819

Gorton, Ian; Chase, Jared M; Wynne, Adam S; Almquist, Justin P; Chappell, Alan R

Scientific applications are often structured as workflows that execute a series of distributed software modules to analyze large data sets. Such workflows are typically constructed using general-purpose scripting languages to coordinate the execution of the various modules and to exchange data sets between them. While such scripts provide a cost-effective approach for simple workflows, as the workflow structure becomes complex and evolves, the scripts quickly become complex and difficult to modify. This makes them a major barrier to easily and quickly deploying new algorithms and exploiting new, scalable hardware platforms. In this paper, we describe the MeDICi Workflow technology that is specifically designed to reduce the complexity of workflow application development, and to efficiently handle data intensive workflow applications. MeDICi integrates standard component-based and service-based technologies, and employs an efficient integration mechanism to ensure large data sets can be efficiently processed. We illustrate the use of MeDICi with a climate data processing example that we have built, and describe some of the new features

• Research Organization: Pacific Northwest National Laboratory (PNNL), Richland, WA (US)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-76RL01830
• OSTI ID: 1011819
• Report Number(s): PNNL-SA-65181
• Country of Publication: United States
• Language: English

Related Subjects

99 GENERAL AND MISCELLANEOUS
ALGORITHMS
CLIMATES
COMPUTER CODES
DATA PROCESSING
PROGRAMMING
middleware
workflow