skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Experiments with In-Transit Processing for Data Intensive Grid Workflows

Abstract

Efficient and robust data streaming and in-transit data manipulations are critical requirements of emerging scientific and engineering application workflows, which are based on seamless interactions and coupling between geographically distributed application components. The overall goal of this research is to address these requirements and develop a data streaming and in-transit data manipulation service. In this paper, we experimentally investigate reactive management strategies for in-transit data manipulation, as well as cooperative endto-end management for wide-area data-streaming and in-transit data manipulation for data-intensive scientific and engineering workflows.

Authors:
 [1];  [1];  [2]
  1. Rutgers University
  2. ORNL
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1054966
DOE Contract Number:
DE-AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: The 8th IEEE/ACM International Conference on Grid Computing (Grid 2007), Austin, TX, USA, 20070919, 20070921
Country of Publication:
United States
Language:
English

Citation Formats

Bhat, Viraj, Parashar, Manish, and Klasky, Scott A. Experiments with In-Transit Processing for Data Intensive Grid Workflows. United States: N. p., 2007. Web.
Bhat, Viraj, Parashar, Manish, & Klasky, Scott A. Experiments with In-Transit Processing for Data Intensive Grid Workflows. United States.
Bhat, Viraj, Parashar, Manish, and Klasky, Scott A. Mon . "Experiments with In-Transit Processing for Data Intensive Grid Workflows". United States. doi:.
@article{osti_1054966,
title = {Experiments with In-Transit Processing for Data Intensive Grid Workflows},
author = {Bhat, Viraj and Parashar, Manish and Klasky, Scott A},
abstractNote = {Efficient and robust data streaming and in-transit data manipulations are critical requirements of emerging scientific and engineering application workflows, which are based on seamless interactions and coupling between geographically distributed application components. The overall goal of this research is to address these requirements and develop a data streaming and in-transit data manipulation service. In this paper, we experimentally investigate reactive management strategies for in-transit data manipulation, as well as cooperative endto-end management for wide-area data-streaming and in-transit data manipulation for data-intensive scientific and engineering workflows.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Mon Jan 01 00:00:00 EST 2007},
month = {Mon Jan 01 00:00:00 EST 2007}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:
  • For the Earth System Grid Federation (ESGF), the ESG-CET team has led international development and delivered a production environment for managing and accessing ultrascale climate data. This production environment includes multiple national and international climate projects (e.g., Couple Model Intercomparison Project, Community Earth System Model), ocean model data (such as the Parallel Ocean Program), observation data (Carbon Dioxide Information and Analysis Center, Atmospheric Infrared Sounder, and so forth), and analysis and visualization tools, all of which serve a diverse community of users. These data holdings and services are distributed across multiple ESG-CET sites (such as LANL, LBNL, LLNL, NCAR, andmore » ORNL) as well as at unfunded partners sites such as the Australian National University National Computational Infrastructure, the British Atmospheric Data Centre, the National Oceanic and Atmospheric Administration Geophysical Fluid Dynamics Laboratory, the Max Planck Institute for Meteorology, the German Climate Computing Centre, and the National Aeronautics and Space Administration Jet Propulsion Laboratory. More recently, ESG-CET has been extending services beyond data-file access and delivery to develop more detailed information products (scientific graphics, animations, etc.), secure binary data-access services (based upon the OPeNDAP protocol), and server-side analysis capabilities. These will allow users to request data subsets transformed through commonly used analysis and intercomparison procedures. As we transition from development activities to production and operations, the ESG-CET team is tasked with making data available to all users seeking to understand, process, extract value from, visualize, and/or communicate it to others. This ongoing effort, though daunting in scope and complexity, will greatly magnify the value of numerical climate model outputs and climate observations for future national and international climate-assessment reports. Continued ESGF progress will result in a production ultrascale data system for empowering scientists who attempt new and exciting data exchanges that could ultimately lead to breakthrough climate-science discoveries.« less
  • The Fermilab Scientific Computing Division and the KISTI Global Science Experimental Data Hub Center have built a prototypical large-scale infrastructure to handle scientific workflows of stakeholders to run on multiple cloud resources. The demonstrations have been in the areas of (a) Data-Intensive Scientific Workflows on Federated Clouds, (b) Interoperability and Federation of Cloud Resources, and (c) Virtual Infrastructure Automation to enable On-Demand Services.